Building dev images

Tue Oct 10 12:27:17 EDT 2006

On Tue, 2006-10-10 at 11:19 -0400, John Richard Moser wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> 
> 
> Dan Williams wrote:
> > On Mon, 2006-10-09 at 22:49 -0400, John Richard Moser wrote:
> > Right now I'm trying to get pilgrim to work and build good dev images on
> > a freshly installed Fedora Core 5 system.  There are a number of things
> > I need to do once I get this working.
> > 
> >   - First off, document it, because it's not on the wiki that I can
> >     find.
> > 
> >   - Get a list of the RPMs used to build the image
> > 
> >> You can look at the build logs for the builds (if they are available) or
> >> you can look at the build logs for the builds you do with the tools.
> >> Yum will tell you exactly what's getting installed into the chroot.
> > 
> 
> Huh.. the bootlog for me tells me a lot of junk was installed but no RPM
> names.  I probably can divine the names out of it.
> 
> For reference
> http://olpc.download.redhat.com/olpc/streams/development/build94/devel_jffs2/build.log
> is what I'm looking at for an example build log; mine is not working,
> it's spitting out broken crap.  :(

Huh?  It's all there; look for "Dependencies Resolved" and everything
below that until "Transaction Summary" is the stuff that's going to be
installed in the chroot.

avahi-tools             i386       0.6.11-6.fc6     olpc_rhbase        56 k

These lines tell you a few things; that avahi-tools, version
0.6.11-6.fc6 from the repo olpc_rhbase is being installed, and that it's
the i386 version of the package.  That should be all you need to get the
RPM name that's being installed:

<name>-<ver/release>.<arch>.rpm

avahi-tools-0.6.11-6.fc6.i386.rpm

> >   - Script converting those RPM names to URIs of SRPMs
> > 
> >> The RPMs come from 3 different repositories: Fedora Core Development,
> >> Fedora Extras Development, and OLPC Development.  You can get the the
> >> SRPM name from which the RPM was generated using a bit of python.
> >> Remember that one SRPM can generate > 1 RPM :)  I can point you to the
> >> right places for the Python code if you'd like.
> > 
> 
> Yeah I know no python (accidentally hacked some a while back but haven't
> touched it since), so pointing me at a chunk of code would be useful.  :)

That would be _so_ much easier to do this with python.  Maybe you could
use Perl too if you know it.

>>> a = "avahi-tools             i386       0.6.11-6.fc6     olpc_rhbase        56 k"
>>> b = a.strip().split()
>>> b
['avahi-tools', 'i386', '0.6.11-6.fc6', 'olpc_rhbase', '56', 'k']
>>> rpm_name = "%s-%s.%s.rpm" % (b[0], b[2], b[1])
>>> rpm_name
'avahi-tools-0.6.11-6.fc6.i386.rpm'
>>> 

For the distro-rebuild stuff with plague, see:

http://people.redhat.com/dcbw/distro-rebuild.py

That's a somewhat hacky script to unpack the SRPM, change the Release #
by adding .1 to the end, and rebuild the SRPM, then enqueue the SRPM in
a plague buildsystem.

> >   - Rebuild those SRPMs (I hear there's this thing called 'mock' that
> >     helps with this en masse)
> > 
> >> Mock builds one SRPM in a chroot.  It does not do builds en-masse, it
> >> builds one SRPM and does it well.  It was decided not to pollute mock
> >> with a ton of policy and options, because different people want to build
> >> a set of RPMs differently.
> > 
> >> Tools like plague will automate the queuing of multiple SRPMs across
> >> different architectures using remote and/or local build hosts.  But mock
> >> knows nothing about networks, nor should it.
> > 
> 
> Cool.  I'll look plague up.
> 
> >> What you want to do is:
> > 
> >> 1) Create a build strategy; do you build glibc first?  do you rebuild
> >> gcc first?  Bootstrapping a distro sucks, but you probably don't want to
> >> do the whole thing.
> > 
> 
> I'll rebuild each package, it doesn't really matter what order because
> I'm not making ABI changes so the libraries will link together fine.
> Bootstrapping isn't necessary to drop in a simple compiler option
> (unless it's -fstack-protector or such).

Ok, your life just got a lot easier.

> >> 2) Once you've got the list of SRPMs you'd like to build and the order
> >> in which you'd like to build them, change the RPM _mock_ macro for
> >> RPM_OPT_FLAGS to include the CFLAGS you want, then create a script that
> >> feeds the SRPMs in that order through mock.  I can dig this up; it means
> >> you change the mock config file (in /etc/mock) for your target and add a
> >> line for your optflags.
> > 
> 
> I'll peek at google and see if there's a manpage.... nope.  I'll look in
> /etc/mock later, yum is busy on that box.

>From the Fedora Extras builders:

[dcbw at hammer1 ~]$ cat /etc/mock/fedora-devel-i386-core.cfg

<snip>

config_opts['macros'] = """
%%vendor Fedora Project
%%distribution Fedora Extras
%%packager Fedora Project <http://bugzilla.redhat.com/bugzilla>
%%_topdir %s/build
%%_rpmfilename   %%%%{NAME}-%%%%{VERSION}-%%%%{RELEASE}.%%%
%{ARCH}.rpm    
""" % config_opts['chroothome']

</snip>

If you look at, for example, /usr/lib/rpm/redhat/macros, you can get an
idea of the optflags you want to override:

optflags: i386 %{__global_cflags} -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables

with your own stuff.  So, in the mock config file for the target for
which you are building your stuff, you'll want to add the
config_opts['macros'] section and do something like:

my_optflags = "-Os -whatever -goreallyfast"
config_opts['macros'] = """
%%optflags %s
""" % my_optflags

> >> 3) When each SRPM is done building, you have two choices.  Do you want
> >> to use the just-built RPM for building other RPMs?  Or do you not care?
> >> Remember that since this is building inside a chroot, you need to get,
> >> for example, /lib/libc-2.4.so from somewhere.  Currently that is pulled
> >> from the Fedora Core Development repository, and it _won't_ be rebuilt
> >> with any of your changes.  If you're just changing CFLAGS, I don't think
> >> this will be a problem.  If you want to do linker changes or something
> >> like that, then it would be a problem.
> > 
> 
> Yeah, it's just CFLAGS.  The linker changes I wanted are standard in
> Fedora Core 6.... and I have FC5.  Eh, it won't matter as long as I
> build 2 different images but maybe I should upgrade my laptop to Rawhide
> anyway (I have no idea how, I'm a debian guy).

It shouldn't matter that you have FC5, because all the building is done
in the chroot, and you're going to pull from the devel repository
anyway.  So the tools & linker that's actually used to rebuild the
packages in the chroot will already be from FC6, since mock will go
fetch them.

> >   - Make a stream to build using a local repository of rebuilt RPMs
> > 
> >> You create a local repository of RPMs using createrepo.  You then point
> >> the pilgrim config file for your stream at this repository, and you
> >> remove any reference to other repositories, because they can pollute
> >> your package pool (unless you do something like bumping the Release # of
> >> the package).  I may have some scripts that can bump the release number
> >> slightly to ensure that the built packages will be "newer" to RPM.
> 
> I can just build everything and stick it in my repo.
> 
> > 
> >   - Get a stream built of mine and of the original RPMs
> > 
> > Right now I'm getting things like:
> > 
> > install-info:  no such file or directory for /usr/share/info/sed.info.gz
> > error: %post(sed-4.1.5-5.fc6.i386) scriptlet failed, exit status 1
> > 
> >> I hope we don't install info at all.  But you will run into this because
> >> the %post, %postun, %pre, and/or %preun sections of the RPMs may be
> >> trying to use utilities that aren't on the system or in your chroot yet.
> >> Most of these are harmless and can be ignored.  If they stop your
> >> install, that's a problem, but means you aren't using the right tool to
> >> install them.
> > 
> > For a lot of things (sed, readline, cpio, cpp, gzip, findutils, tar...),
> > not sure why.  Last time I did this I got a bunch of empty directories
> > that should have contained images.
> > 
> >> Some of these, we plain may not install.  It's quite hard to imagine why
> >> we would include 'cpp' by default.  I know it's used for more than
> >> software development, but chances are we don't care about what uses it
> >> in the default images.
> > 
> > I figure once I get the building to actually work, I'll move on to
> > figuring out how to get a list of RPMs used, and then turn those into
> > URLs for SRPMs, and work from there.  Not sure where to start with this
> > though, the build.log has some stuff but it's cryptic and may be a
> > little challenging to parse.
> > 
> >> Rebuilding a bunch of stuff is hard work, and it's manual work, because
> >> unless you're on a source-based distro like Gentoo or LFS, you never do
> >> this, your distro release team hits the rebuild switch when it's
> >> required.  And they have all the infrastructure set up already.
> > 
> 
> bash scripts.  The reason Linux pwns Windows.  :)
> 
> > Is there any documentation at all for this?  I know I'm not there yet;
> > but I'm primarily concerned with figuring out exactly what RPMs are used
> > (so I can build proper RPMs from SRPMs) and switching over to a local
> > repo (so I can build an image without an -O2 enabled optimization that I
> > believe may be problematic on the Geode and do some tests).
> > 
> >> You can use a tool like plague
> >> (http://www.fedoraproject.org/wiki/Projects/Plague ) to automate the
> >> rebuild process, even across multiple machines.  It depends on your
> >> needs, if you feel comfortable scripting the mock rebuild process
> >> yourself, or using a tool like plague (which runs the Fedora Extras
> >> buildsystem http://buildsys.fedoraproject.org/build-status/ ) which
> >> requires some initial setup.
> > 
> 
> mmm... scripting it myself may be less to learn.
> 
> Aside, I've got an ext3 .img and a jffs2 tree now, but no jffs2 .img.  Hmm.
> 
> >> Dan
> > 
> 
> - --
>     We will enslave their women, eat their children and rape their
>     cattle!
>                   -- Bosc, Evil alien overlord from the fifth dimension
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.3 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
> 
> iQIVAwUBRSu6Ggs1xW0HCTEFAQLm/RAAkW0EO62TAwS3R7l5atCPkvttHFKlf1ut
> CD3Oh69KwPwscjLwdHF47SV/cccEzC64ygWiADPGsjqsZ/mSJS06hybT3rC6T7TE
> Ns3EcWxBtTuKgFjK5mx1EMzDZ11ejEByWAf+kr4bJCEByOeWZrSPj6C8gC0qDtbk
> YFWLcWAb3VadSD9+YOdp7TITF+YcVRHOWk+GrfBAFyYmobaCRdiaLQdnVLQpqUsu
> VI5IeF3wgcjpSuZ94jdXn2Vxfo/yMN4EhAo0EM6vjLgEE4hROGioU+p0RggaD+ec
> DTWViwavTYCi1yNivv5ia3SkO67Kr5x6Bb11gFN2qTEQXIjIIZy4FNHb/diQTvsE
> u/j8LtQ8t1+sNB3wOPdPQjz9C3L+TKa31etsYH4VMfLlPDW8cH/OOOEHiCVElHNg
> xx+l/B2KJlR2x0K/TJmlY+eVke7oNQGhhqnUoqAH40edqMbs1XKvXi+a/h4Nw2Vb
> 8BR40NfQF6bdsqOIgsjkCl4FQEgKgDOsvY83GYxMeb2OBWw7z3YSBFKUtqbHJqzr
> eSvjuVs75uzwIQ9eHVZUPOH61mGqbpjgsbgG84jZNkBZDnWiJeUElXViVUrf8hKI
> n3lOus2EBylbmHmW5YDxgrRxzEqMUiiXZnL0V1A+XyGnBiB0+J/jUgHGqNv8ut80
> foCG0CKxgzU=
> =iEn4
> -----END PGP SIGNATURE-----