[Fwd: Need Help on creating SMALL/Delta O.S. Builds] (schedule related)

[Carl-Daniel Hailfinger]

John (J5) Palmieri wrote:
> On Fri, 2007-01-05 at 20:23 +0000, William Bradley wrote:
>> 24 MB is still pretty daunting at the bit rates we have been seeing lately.
>>
>> If C-D can beat that significantly, it would be worth doing.  As the
>> number of laptops increases, the value of such a capability will only
>> increase.
> This is just stopgap for right now.  We can get better I think by doing
> diffs of the trees themselves.  That is why we generate them and keep

Yeah, that would be the first step towards better compression. Depending
on the structure of jffs2, this might give us another factor of 2.

> them around.  The idea is to do a bdiff on them inorder to provide
> updates to base systems.  Of course there are issue we need to work out
> like a user changing configuration files.

These problems (config files etc.) have already been solved by the
deltarpm package which is available at for openSUSE and Fedora.

> If C-D can come up with
> something in the meantime, that would be great.  I'm doing a build with
> a smaller changeset tonight.  We will see if the diff still remains
> high.

Assuming that JFFS2 file system creation is deterministic (in the weak
sense of the word: "no pseudorandom/time dependent stuff") and that
{m,a,c}time of all file types (including symlinks (problem!)) on JFFS2
are either settable or not stored on disk, two JFFS2 file systems
created from the same file tree (with the same file insertion order
and readdir order) should be bit identical.

The challenge is to fix the following points:
* Linux can't set {m,a,c}time on symlinks
* {a,c}time
* timestamps in the file system itself (fs creation time...)
* if fs creation depends on readdir order of the source fs, you are
  screwed because the readdir order depends on the used fs type and
  sometimes directory name and the phase of the moon
* the published images should be created through the delta processs
  to be sure that the downloadable image is the same as a locally
  recreated image

If you want to get the size down even further, you have to change
the build process itself and maybe deltarpm.

Suggestions:
* use "--preserve-timestamps" for the install command in all build
  scripts
* do not precompress any content, but if you do, make sure that you
  do not include any timestamps in the compressed file (use the
  "--no-name" option for gzip)
* change the compression algorithm for the binary diff to lzma

Regards,
Carl-Daniel
-- 
http://www.hailfinger.org/