Building dev images

Tue Oct 10 10:43:28 EDT 2006

On Mon, 2006-10-09 at 22:49 -0400, John Richard Moser wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Right now I'm trying to get pilgrim to work and build good dev images on
> a freshly installed Fedora Core 5 system.  There are a number of things
> I need to do once I get this working.
> 
>   - First off, document it, because it's not on the wiki that I can
>     find.
> 
>   - Get a list of the RPMs used to build the image

You can look at the build logs for the builds (if they are available) or
you can look at the build logs for the builds you do with the tools.
Yum will tell you exactly what's getting installed into the chroot.

>   - Script converting those RPM names to URIs of SRPMs

The RPMs come from 3 different repositories: Fedora Core Development,
Fedora Extras Development, and OLPC Development.  You can get the the
SRPM name from which the RPM was generated using a bit of python.
Remember that one SRPM can generate > 1 RPM :)  I can point you to the
right places for the Python code if you'd like.

>   - Rebuild those SRPMs (I hear there's this thing called 'mock' that
>     helps with this en masse)

Mock builds one SRPM in a chroot.  It does not do builds en-masse, it
builds one SRPM and does it well.  It was decided not to pollute mock
with a ton of policy and options, because different people want to build
a set of RPMs differently.

Tools like plague will automate the queuing of multiple SRPMs across
different architectures using remote and/or local build hosts.  But mock
knows nothing about networks, nor should it.

What you want to do is:

1) Create a build strategy; do you build glibc first?  do you rebuild
gcc first?  Bootstrapping a distro sucks, but you probably don't want to
do the whole thing.

2) Once you've got the list of SRPMs you'd like to build and the order
in which you'd like to build them, change the RPM _mock_ macro for
RPM_OPT_FLAGS to include the CFLAGS you want, then create a script that
feeds the SRPMs in that order through mock.  I can dig this up; it means
you change the mock config file (in /etc/mock) for your target and add a
line for your optflags.

3) When each SRPM is done building, you have two choices.  Do you want
to use the just-built RPM for building other RPMs?  Or do you not care?
Remember that since this is building inside a chroot, you need to get,
for example, /lib/libc-2.4.so from somewhere.  Currently that is pulled
from the Fedora Core Development repository, and it _won't_ be rebuilt
with any of your changes.  If you're just changing CFLAGS, I don't think
this will be a problem.  If you want to do linker changes or something
like that, then it would be a problem.

>   - Make a stream to build using a local repository of rebuilt RPMs

You create a local repository of RPMs using createrepo.  You then point
the pilgrim config file for your stream at this repository, and you
remove any reference to other repositories, because they can pollute
your package pool (unless you do something like bumping the Release # of
the package).  I may have some scripts that can bump the release number
slightly to ensure that the built packages will be "newer" to RPM.

>   - Get a stream built of mine and of the original RPMs
> 
> Right now I'm getting things like:
> 
> install-info:  no such file or directory for /usr/share/info/sed.info.gz
> error: %post(sed-4.1.5-5.fc6.i386) scriptlet failed, exit status 1

I hope we don't install info at all.  But you will run into this because
the %post, %postun, %pre, and/or %preun sections of the RPMs may be
trying to use utilities that aren't on the system or in your chroot yet.
Most of these are harmless and can be ignored.  If they stop your
install, that's a problem, but means you aren't using the right tool to
install them.

> For a lot of things (sed, readline, cpio, cpp, gzip, findutils, tar...),
> not sure why.  Last time I did this I got a bunch of empty directories
> that should have contained images.

Some of these, we plain may not install.  It's quite hard to imagine why
we would include 'cpp' by default.  I know it's used for more than
software development, but chances are we don't care about what uses it
in the default images.

> I figure once I get the building to actually work, I'll move on to
> figuring out how to get a list of RPMs used, and then turn those into
> URLs for SRPMs, and work from there.  Not sure where to start with this
> though, the build.log has some stuff but it's cryptic and may be a
> little challenging to parse.

Rebuilding a bunch of stuff is hard work, and it's manual work, because
unless you're on a source-based distro like Gentoo or LFS, you never do
this, your distro release team hits the rebuild switch when it's
required.  And they have all the infrastructure set up already.

> Is there any documentation at all for this?  I know I'm not there yet;
> but I'm primarily concerned with figuring out exactly what RPMs are used
> (so I can build proper RPMs from SRPMs) and switching over to a local
> repo (so I can build an image without an -O2 enabled optimization that I
> believe may be problematic on the Geode and do some tests).

You can use a tool like plague
(http://www.fedoraproject.org/wiki/Projects/Plague ) to automate the
rebuild process, even across multiple machines.  It depends on your
needs, if you feel comfortable scripting the mock rebuild process
yourself, or using a tool like plague (which runs the Fedora Extras
buildsystem http://buildsys.fedoraproject.org/build-status/ ) which
requires some initial setup.

Dan

> It'd also be interesting to me to get bootchart on this thing, possibly
> in the devel images.  The chart produced is only 173K so a hack to get
> it spit into a tmpfs (i.e. to avoid writing 3 extra blocks to NAND on
> every boot) would be cool but that's getting into flaming optimizations.
> - --
> All content of all messages exchanged herein are left in the
> Public Domain, unless otherwise explicitly stated.
> 
>     Creative brains are a valuable, limited resource. They shouldn't be
>     wasted on re-inventing the wheel when there are so many fascinating
>     new problems waiting out there.
>                                                  -- Eric Steven Raymond
> 
>     We will enslave their women, eat their children and rape their
>     cattle!
>                   -- Bosc, Evil alien overlord from the fifth dimension
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.3 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
> 
> iQIVAwUBRSsKLgs1xW0HCTEFAQKdpBAAnEkDKWeNwPnG22KAc8PPe/AbCBpHBBQA
> zNMSZ08rWsWNRm3HqNq7N8ToqCRovwU4/8QFnCKKir3ZFtGaKfsme/OoPqx0Xoz1
> gt10jmYQSdEz/+mFGJIAJSIANm+7KArWZwnqFV9USaJO706HzddMH9vD49yLDy05
> EHk7puk1jot61h8NLQgDOxw9HxJjgtRtCFHBBobUno/g/SC9z/h64g6D02Ln5iQH
> 50IFinUQZufjJ+5Z5rIYht4EGrw7t2vkDd9a8B8D5RNzgr2Oog5wTnwGq8YPa6xz
> tVspKkmXp2VCXYg3I1AFl18FlgStwkfwEpSMtdtuEA7k8FgHYdJ9soVsG7FJMC/3
> yJDhDzFXpnkgjmTTyZMYe9wxCL8KOKFof6mEmgCwMaw51wGa2sjBbZJ2D0gQcuLF
> lgvh0oUDfhXxM1J0OGA4PvdQkfXFtHKxOBRJQw0mG/fwvMfvvxIq/+9ZeVwCBDTO
> Hte17cq+N4T009Mjn/vHFYvoE8su8yqsdxK5egG4rYIpsFButEJNpVQkLFCo672Q
> UKygYJqD1iF2alGhG6PqkOT0hd3idat3TeC2vUnPwTC/XBTY266uDxjg+X+nKjg1
> 3RP7wUROyC1dtFZG+9vPv4Pw7djSsMAgnB1RTOKcQtQOaM3r8HpH38c5I2v6bC2H
> mk0wIhJPMNo=
> =Qqov
> -----END PGP SIGNATURE-----
> _______________________________________________
> Devel mailing list
> Devel at laptop.org
> http://mailman.laptop.org/mailman/listinfo/devel