Building dev images

[John Richard Moser]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



Dan Williams wrote:
> On Tue, 2006-10-10 at 11:19 -0400, John Richard Moser wrote:
> 
> 
> Dan Williams wrote:
>>>> On Mon, 2006-10-09 at 22:49 -0400, John Richard Moser wrote:
>>>> Right now I'm trying to get pilgrim to work and build good dev images on
>>>> a freshly installed Fedora Core 5 system.  There are a number of things
>>>> I need to do once I get this working.
>>>>
>>>>   - First off, document it, because it's not on the wiki that I can
>>>>     find.
>>>>
>>>>   - Get a list of the RPMs used to build the image
>>>>
>>>>> You can look at the build logs for the builds (if they are available) or
>>>>> you can look at the build logs for the builds you do with the tools.
>>>>> Yum will tell you exactly what's getting installed into the chroot.
> Huh.. the bootlog for me tells me a lot of junk was installed but no RPM
> names.  I probably can divine the names out of it.
> 
> For reference
> http://olpc.download.redhat.com/olpc/streams/development/build94/devel_jffs2/build.log
> is what I'm looking at for an example build log; mine is not working,
> it's spitting out broken crap.  :(
> 
>> Huh?  It's all there; look for "Dependencies Resolved" and everything
>> below that until "Transaction Summary" is the stuff that's going to be
>> installed in the chroot.
> 
>> avahi-tools             i386       0.6.11-6.fc6     olpc_rhbase        56 k
> 
>> These lines tell you a few things; that avahi-tools, version
>> 0.6.11-6.fc6 from the repo olpc_rhbase is being installed, and that it's
>> the i386 version of the package.  That should be all you need to get the
>> RPM name that's being installed:
> 
>> <name>-<ver/release>.<arch>.rpm
> 
>> avahi-tools-0.6.11-6.fc6.i386.rpm
> 

Ah, yes thanks.  I'm a debian guy, I don't know how to abstractly
construct RPM names from that stuff :)

>>>>   - Script converting those RPM names to URIs of SRPMs
>>>>
>>>>> The RPMs come from 3 different repositories: Fedora Core Development,
>>>>> Fedora Extras Development, and OLPC Development.  You can get the the
>>>>> SRPM name from which the RPM was generated using a bit of python.
>>>>> Remember that one SRPM can generate > 1 RPM :)  I can point you to the
>>>>> right places for the Python code if you'd like.
> Yeah I know no python (accidentally hacked some a while back but haven't
> touched it since), so pointing me at a chunk of code would be useful.  :)
> 
>> That would be _so_ much easier to do this with python.  Maybe you could
>> use Perl too if you know it.
> 
>>>>> a = "avahi-tools             i386       0.6.11-6.fc6     olpc_rhbase        56 k"
>>>>> b = a.strip().split()
>>>>> b
>> ['avahi-tools', 'i386', '0.6.11-6.fc6', 'olpc_rhbase', '56', 'k']
>>>>> rpm_name = "%s-%s.%s.rpm" % (b[0], b[2], b[1])
>>>>> rpm_name
>> 'avahi-tools-0.6.11-6.fc6.i386.rpm'
> 
>> For the distro-rebuild stuff with plague, see:
> 
>> http://people.redhat.com/dcbw/distro-rebuild.py
> 
>> That's a somewhat hacky script to unpack the SRPM, change the Release #
>> by adding .1 to the end, and rebuild the SRPM, then enqueue the SRPM in
>> a plague buildsystem.
> 
>>>>   - Rebuild those SRPMs (I hear there's this thing called 'mock' that
>>>>     helps with this en masse)
>>>>
>>>>> Mock builds one SRPM in a chroot.  It does not do builds en-masse, it
>>>>> builds one SRPM and does it well.  It was decided not to pollute mock
>>>>> with a ton of policy and options, because different people want to build
>>>>> a set of RPMs differently.
>>>>> Tools like plague will automate the queuing of multiple SRPMs across
>>>>> different architectures using remote and/or local build hosts.  But mock
>>>>> knows nothing about networks, nor should it.
> Cool.  I'll look plague up.
> 
>>>>> What you want to do is:
>>>>> 1) Create a build strategy; do you build glibc first?  do you rebuild
>>>>> gcc first?  Bootstrapping a distro sucks, but you probably don't want to
>>>>> do the whole thing.
> I'll rebuild each package, it doesn't really matter what order because
> I'm not making ABI changes so the libraries will link together fine.
> Bootstrapping isn't necessary to drop in a simple compiler option
> (unless it's -fstack-protector or such).
> 
>> Ok, your life just got a lot easier.
> 
>>>>> 2) Once you've got the list of SRPMs you'd like to build and the order
>>>>> in which you'd like to build them, change the RPM _mock_ macro for
>>>>> RPM_OPT_FLAGS to include the CFLAGS you want, then create a script that
>>>>> feeds the SRPMs in that order through mock.  I can dig this up; it means
>>>>> you change the mock config file (in /etc/mock) for your target and add a
>>>>> line for your optflags.
> I'll peek at google and see if there's a manpage.... nope.  I'll look in
> /etc/mock later, yum is busy on that box.
> 
>>>From the Fedora Extras builders:
> 
>> [dcbw at hammer1 ~]$ cat /etc/mock/fedora-devel-i386-core.cfg
> 
>> <snip>
> 
>> config_opts['macros'] = """
>> %%vendor Fedora Project
>> %%distribution Fedora Extras
>> %%packager Fedora Project <http://bugzilla.redhat.com/bugzilla>
>> %%_topdir %s/build
>> %%_rpmfilename   %%%%{NAME}-%%%%{VERSION}-%%%%{RELEASE}.%%%
>> %{ARCH}.rpm    
>> """ % config_opts['chroothome']
> 
>> </snip>
> 
>> If you look at, for example, /usr/lib/rpm/redhat/macros, you can get an
>> idea of the optflags you want to override:
> 
>> optflags: i386 %{__global_cflags} -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables
> 
>> with your own stuff.  So, in the mock config file for the target for
>> which you are building your stuff, you'll want to add the
>> config_opts['macros'] section and do something like:
> 
>> my_optflags = "-Os -whatever -goreallyfast"
>> config_opts['macros'] = """
>> %%optflags %s
>> """ % my_optflags


...... kay, I think I got that.  We'll see when I get there :)


> 
>>>>> 3) When each SRPM is done building, you have two choices.  Do you want
>>>>> to use the just-built RPM for building other RPMs?  Or do you not care?
>>>>> Remember that since this is building inside a chroot, you need to get,
>>>>> for example, /lib/libc-2.4.so from somewhere.  Currently that is pulled
>>>>> from the Fedora Core Development repository, and it _won't_ be rebuilt
>>>>> with any of your changes.  If you're just changing CFLAGS, I don't think
>>>>> this will be a problem.  If you want to do linker changes or something
>>>>> like that, then it would be a problem.
> Yeah, it's just CFLAGS.  The linker changes I wanted are standard in
> Fedora Core 6.... and I have FC5.  Eh, it won't matter as long as I
> build 2 different images but maybe I should upgrade my laptop to Rawhide
> anyway (I have no idea how, I'm a debian guy).
> 
>> It shouldn't matter that you have FC5, because all the building is done
>> in the chroot, and you're going to pull from the devel repository
>> anyway.  So the tools & linker that's actually used to rebuild the
>> packages in the chroot will already be from FC6, since mock will go
>> fetch them.
> 

Sweet.  That makes life much much easier.


>>>>   - Make a stream to build using a local repository of rebuilt RPMs
>>>>
>>>>> You create a local repository of RPMs using createrepo.  You then point
>>>>> the pilgrim config file for your stream at this repository, and you
>>>>> remove any reference to other repositories, because they can pollute
>>>>> your package pool (unless you do something like bumping the Release # of
>>>>> the package).  I may have some scripts that can bump the release number
>>>>> slightly to ensure that the built packages will be "newer" to RPM.
> I can just build everything and stick it in my repo.
> 
>>>>   - Get a stream built of mine and of the original RPMs
>>>>
>>>> Right now I'm getting things like:
>>>>
>>>> install-info:  no such file or directory for /usr/share/info/sed.info.gz
>>>> error: %post(sed-4.1.5-5.fc6.i386) scriptlet failed, exit status 1
>>>>
>>>>> I hope we don't install info at all.  But you will run into this because
>>>>> the %post, %postun, %pre, and/or %preun sections of the RPMs may be
>>>>> trying to use utilities that aren't on the system or in your chroot yet.
>>>>> Most of these are harmless and can be ignored.  If they stop your
>>>>> install, that's a problem, but means you aren't using the right tool to
>>>>> install them.
>>>> For a lot of things (sed, readline, cpio, cpp, gzip, findutils, tar...),
>>>> not sure why.  Last time I did this I got a bunch of empty directories
>>>> that should have contained images.
>>>>
>>>>> Some of these, we plain may not install.  It's quite hard to imagine why
>>>>> we would include 'cpp' by default.  I know it's used for more than
>>>>> software development, but chances are we don't care about what uses it
>>>>> in the default images.
>>>> I figure once I get the building to actually work, I'll move on to
>>>> figuring out how to get a list of RPMs used, and then turn those into
>>>> URLs for SRPMs, and work from there.  Not sure where to start with this
>>>> though, the build.log has some stuff but it's cryptic and may be a
>>>> little challenging to parse.
>>>>
>>>>> Rebuilding a bunch of stuff is hard work, and it's manual work, because
>>>>> unless you're on a source-based distro like Gentoo or LFS, you never do
>>>>> this, your distro release team hits the rebuild switch when it's
>>>>> required.  And they have all the infrastructure set up already.
> bash scripts.  The reason Linux pwns Windows.  :)
> 
>>>> Is there any documentation at all for this?  I know I'm not there yet;
>>>> but I'm primarily concerned with figuring out exactly what RPMs are used
>>>> (so I can build proper RPMs from SRPMs) and switching over to a local
>>>> repo (so I can build an image without an -O2 enabled optimization that I
>>>> believe may be problematic on the Geode and do some tests).
>>>>
>>>>> You can use a tool like plague
>>>>> (http://www.fedoraproject.org/wiki/Projects/Plague ) to automate the
>>>>> rebuild process, even across multiple machines.  It depends on your
>>>>> needs, if you feel comfortable scripting the mock rebuild process
>>>>> yourself, or using a tool like plague (which runs the Fedora Extras
>>>>> buildsystem http://buildsys.fedoraproject.org/build-status/ ) which
>>>>> requires some initial setup.
> mmm... scripting it myself may be less to learn.
> 
> Aside, I've got an ext3 .img and a jffs2 tree now, but no jffs2 .img.  Hmm.
> 
>>>>> Dan

- --
    We will enslave their women, eat their children and rape their
    cattle!
                  -- Bosc, Evil alien overlord from the fifth dimension
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQIVAwUBRSv9pAs1xW0HCTEFAQL22w//ZEJvmvThDspTy9kakPbEV4OMOZMa/QEn
x2gw6UK9OsXAWkGvwpOT6m9CzOMVy8DUawFZm2FsgRl5OAW6+fcoJihDsKdhRgZ1
qoRMqBEAb1B2YAxUAponekZxzk3XyvdqIog/thGp4fpZKY8xJu/VPfr/w9Kb/ZLJ
g7qEIHnhT7UQ0fo9ul9vYcr4B9thkwW+U8FGl9Jv081ZCV3emedcc+bQfrcWH3cv
i6qzL+KOLJTno1LRHrrZFhaqbiWCYqzDpj9/wxSRWtnjKoZNtkT5ZfX+UDP5fqEW
ffwXW1U/fP0BJOdzM9VarqjWW0syXRq2g81NJQveobc8I15MIQkiqXqBrAHFeEE2
997UweOgKaafrTVrXupPWtuEmikpoKdVRVIR2UkOhSaB8HjGd4Yzrlp6KCO/TmYG
8vmQU+2Wi8/0sN55Zn31XJpA1lm/LAb9SNE8pZbor/e/0BnsXejso4gCBkbM7EZV
y9aDYSeizYVI87n6UL5O/SjQ/xJUkBhlYpVxQ9N6Sd9QR6BbsVapYnhVQ+PGKMqj
rBp+gr54NFIhOT2y0LLFCG+a6ImLwmsjONB+0vU0Gnz55S/TD+LV4rPGvUMhd8Df
LMVJKUhC9fIYXb6IXqsX6blZpVUYekiFnjo64PktHVTcagFFZBcmSE9X46dVDsh8
SlvSy4yVcgk=
=cLUz
-----END PGP SIGNATURE-----