Building dev images

Tue Oct 10 11:19:56 EDT 2006

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dan Williams wrote:
> On Mon, 2006-10-09 at 22:49 -0400, John Richard Moser wrote:
> Right now I'm trying to get pilgrim to work and build good dev images on
> a freshly installed Fedora Core 5 system.  There are a number of things
> I need to do once I get this working.
> 
>   - First off, document it, because it's not on the wiki that I can
>     find.
> 
>   - Get a list of the RPMs used to build the image
> 
>> You can look at the build logs for the builds (if they are available) or
>> you can look at the build logs for the builds you do with the tools.
>> Yum will tell you exactly what's getting installed into the chroot.
> 

Huh.. the bootlog for me tells me a lot of junk was installed but no RPM
names.  I probably can divine the names out of it.

For reference
http://olpc.download.redhat.com/olpc/streams/development/build94/devel_jffs2/build.log
is what I'm looking at for an example build log; mine is not working,
it's spitting out broken crap.  :(

>   - Script converting those RPM names to URIs of SRPMs
> 
>> The RPMs come from 3 different repositories: Fedora Core Development,
>> Fedora Extras Development, and OLPC Development.  You can get the the
>> SRPM name from which the RPM was generated using a bit of python.
>> Remember that one SRPM can generate > 1 RPM :)  I can point you to the
>> right places for the Python code if you'd like.
> 

Yeah I know no python (accidentally hacked some a while back but haven't
touched it since), so pointing me at a chunk of code would be useful.  :)

>   - Rebuild those SRPMs (I hear there's this thing called 'mock' that
>     helps with this en masse)
> 
>> Mock builds one SRPM in a chroot.  It does not do builds en-masse, it
>> builds one SRPM and does it well.  It was decided not to pollute mock
>> with a ton of policy and options, because different people want to build
>> a set of RPMs differently.
> 
>> Tools like plague will automate the queuing of multiple SRPMs across
>> different architectures using remote and/or local build hosts.  But mock
>> knows nothing about networks, nor should it.
> 

Cool.  I'll look plague up.

>> What you want to do is:
> 
>> 1) Create a build strategy; do you build glibc first?  do you rebuild
>> gcc first?  Bootstrapping a distro sucks, but you probably don't want to
>> do the whole thing.
> 

I'll rebuild each package, it doesn't really matter what order because
I'm not making ABI changes so the libraries will link together fine.
Bootstrapping isn't necessary to drop in a simple compiler option
(unless it's -fstack-protector or such).

>> 2) Once you've got the list of SRPMs you'd like to build and the order
>> in which you'd like to build them, change the RPM _mock_ macro for
>> RPM_OPT_FLAGS to include the CFLAGS you want, then create a script that
>> feeds the SRPMs in that order through mock.  I can dig this up; it means
>> you change the mock config file (in /etc/mock) for your target and add a
>> line for your optflags.
> 

I'll peek at google and see if there's a manpage.... nope.  I'll look in
/etc/mock later, yum is busy on that box.

>> 3) When each SRPM is done building, you have two choices.  Do you want
>> to use the just-built RPM for building other RPMs?  Or do you not care?
>> Remember that since this is building inside a chroot, you need to get,
>> for example, /lib/libc-2.4.so from somewhere.  Currently that is pulled
>> from the Fedora Core Development repository, and it _won't_ be rebuilt
>> with any of your changes.  If you're just changing CFLAGS, I don't think
>> this will be a problem.  If you want to do linker changes or something
>> like that, then it would be a problem.
> 

Yeah, it's just CFLAGS.  The linker changes I wanted are standard in
Fedora Core 6.... and I have FC5.  Eh, it won't matter as long as I
build 2 different images but maybe I should upgrade my laptop to Rawhide
anyway (I have no idea how, I'm a debian guy).

>   - Make a stream to build using a local repository of rebuilt RPMs
> 
>> You create a local repository of RPMs using createrepo.  You then point
>> the pilgrim config file for your stream at this repository, and you
>> remove any reference to other repositories, because they can pollute
>> your package pool (unless you do something like bumping the Release # of
>> the package).  I may have some scripts that can bump the release number
>> slightly to ensure that the built packages will be "newer" to RPM.

I can just build everything and stick it in my repo.

> 
>   - Get a stream built of mine and of the original RPMs
> 
> Right now I'm getting things like:
> 
> install-info:  no such file or directory for /usr/share/info/sed.info.gz
> error: %post(sed-4.1.5-5.fc6.i386) scriptlet failed, exit status 1
> 
>> I hope we don't install info at all.  But you will run into this because
>> the %post, %postun, %pre, and/or %preun sections of the RPMs may be
>> trying to use utilities that aren't on the system or in your chroot yet.
>> Most of these are harmless and can be ignored.  If they stop your
>> install, that's a problem, but means you aren't using the right tool to
>> install them.
> 
> For a lot of things (sed, readline, cpio, cpp, gzip, findutils, tar...),
> not sure why.  Last time I did this I got a bunch of empty directories
> that should have contained images.
> 
>> Some of these, we plain may not install.  It's quite hard to imagine why
>> we would include 'cpp' by default.  I know it's used for more than
>> software development, but chances are we don't care about what uses it
>> in the default images.
> 
> I figure once I get the building to actually work, I'll move on to
> figuring out how to get a list of RPMs used, and then turn those into
> URLs for SRPMs, and work from there.  Not sure where to start with this
> though, the build.log has some stuff but it's cryptic and may be a
> little challenging to parse.
> 
>> Rebuilding a bunch of stuff is hard work, and it's manual work, because
>> unless you're on a source-based distro like Gentoo or LFS, you never do
>> this, your distro release team hits the rebuild switch when it's
>> required.  And they have all the infrastructure set up already.
> 

bash scripts.  The reason Linux pwns Windows.  :)

> Is there any documentation at all for this?  I know I'm not there yet;
> but I'm primarily concerned with figuring out exactly what RPMs are used
> (so I can build proper RPMs from SRPMs) and switching over to a local
> repo (so I can build an image without an -O2 enabled optimization that I
> believe may be problematic on the Geode and do some tests).
> 
>> You can use a tool like plague
>> (http://www.fedoraproject.org/wiki/Projects/Plague ) to automate the
>> rebuild process, even across multiple machines.  It depends on your
>> needs, if you feel comfortable scripting the mock rebuild process
>> yourself, or using a tool like plague (which runs the Fedora Extras
>> buildsystem http://buildsys.fedoraproject.org/build-status/ ) which
>> requires some initial setup.
> 

mmm... scripting it myself may be less to learn.

Aside, I've got an ext3 .img and a jffs2 tree now, but no jffs2 .img.  Hmm.

>> Dan
> 

- --
    We will enslave their women, eat their children and rape their
    cattle!
                  -- Bosc, Evil alien overlord from the fifth dimension
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQIVAwUBRSu6Ggs1xW0HCTEFAQLm/RAAkW0EO62TAwS3R7l5atCPkvttHFKlf1ut
CD3Oh69KwPwscjLwdHF47SV/cccEzC64ygWiADPGsjqsZ/mSJS06hybT3rC6T7TE
Ns3EcWxBtTuKgFjK5mx1EMzDZ11ejEByWAf+kr4bJCEByOeWZrSPj6C8gC0qDtbk
YFWLcWAb3VadSD9+YOdp7TITF+YcVRHOWk+GrfBAFyYmobaCRdiaLQdnVLQpqUsu
VI5IeF3wgcjpSuZ94jdXn2Vxfo/yMN4EhAo0EM6vjLgEE4hROGioU+p0RggaD+ec
DTWViwavTYCi1yNivv5ia3SkO67Kr5x6Bb11gFN2qTEQXIjIIZy4FNHb/diQTvsE
u/j8LtQ8t1+sNB3wOPdPQjz9C3L+TKa31etsYH4VMfLlPDW8cH/OOOEHiCVElHNg
xx+l/B2KJlR2x0K/TJmlY+eVke7oNQGhhqnUoqAH40edqMbs1XKvXi+a/h4Nw2Vb
8BR40NfQF6bdsqOIgsjkCl4FQEgKgDOsvY83GYxMeb2OBWw7z3YSBFKUtqbHJqzr
eSvjuVs75uzwIQ9eHVZUPOH61mGqbpjgsbgG84jZNkBZDnWiJeUElXViVUrf8hKI
n3lOus2EBylbmHmW5YDxgrRxzEqMUiiXZnL0V1A+XyGnBiB0+J/jUgHGqNv8ut80
foCG0CKxgzU=
=iEn4
-----END PGP SIGNATURE-----