equivalent copy-nand interface ideas for XO-1.5

[Daniel Drake]

It may be a little early for this discussion but I wanted to write down
some thoughts...

The XO-1.5 has an ATA-style flash interface, we no longer have raw flash
access (this isn't going to change now, right?). This means that we have
to move away from jffs2 to a "normal" filesystem such as ext2.

The copy-nand technology present in OFW on the XO-1 is invaluable to
deployments (allowing for USB-based reflash, nandblaster, etc) but is
implemented around the features of having raw flash access, and also
from OFW having a limited understanding of JFFS2.

We need an equivalent for XO-1.5. And the same high-level properties are
desirable:
1. The size of the source image file should reflect the size of the data
within it (i.e. we may have a 4GB disk, but if we want to flash a 100mb
image then the source image file should be approximately 100mb rather
than 4gb in size).
2. Openfirmware should only have to write to the blocks on disk where
data is used, rather than having to write a whole 4gb of data every
time.


It's tricky because the different formats (and disk access methodology)
mean that we don't have the same options available to us. For example,
how do you make an image file of an 4GB ext2 filesystem (with 100mb
used) that isn't 4GB in size?


I've been thinking about this a little...

One option is to use sparse files. To make a 100mb sparse file:
	dd if=/dev/zero of=outfile bs=1M count=0 seek=100
This file appears 100mb in size but uses no disk space at all. Once you
start writing to it, the blocks that you write to will be allocated and
saved on disk (using the usual amount of space) but the empty parts of
the file will still not actually consume disk space. (You could now run
mke2fs on this file to turn it into a loop-mountable filesystem image,
initially only occupying 3.6mb of "real" disk space through the ext2
formatting process.)

However, problems are encountered when you try and distribute this file
over mediums that don't understand sparse files. For example if I put
that 100mb empty sparse file on a webserver and download it over HTTP,
100mb of data will be transferred over the network and the resultant
empty file will occupy the full 100mb of disk space even though the
source was empty. Same if I copy it to a FAT filesystem.

We noticed that gzipping a sparse file is quite efficient, the
compression of the unused blocks uses very little space in the resultant
gzip output, but the problem here is that you lose the sparseness (when
you gunzip, the file is not sparse and stores all the blocks on disk,
even if they were originally unused).

I just found a better option: tar
tar can store sparse-files efficiently; it basically just stores the
offsets of the file where there is actually data, and the data itself.
	tar -cSf outfile.tar outfile
Then we can compress the actual data
	zip outfile.zip outfile.tar

Now my 100mb sparse file (which used 0 disk blocks) is represented as a
file of 275 bytes which I can easily distribute over http or FAT, and if
I extract it then the result will also use 0 disk blocks because the tar
archive encodes that information in detail. So, that's desirable
property #1 implemented in a way where the sparse-ness is not lost.


Now for the flashing part... OFW already understands .zip but would need
to grow a basic understanding of .tar. Specifically it would have to
locate the one file that we'd put in the tar (the disk image), and it
would have to understand the sparse-ness of it. Then it would write the
appropriate data to the relevant disk sectors.

This would be filesystem-independent (OFW doesn't need to understand the
format of the disk image inside the .tar at all) and only writes the
used blocks of data to disk, without touching the others.
But one disadvantage is that it doesn't touch the other blocks, meaning
that data from the previous installation might still be accessible if
someone starts reading fs-unused blocks -- is this a security/privacy
issue?
I am assuming we don't have an ultra-fast way of erasing blocks like we
did with raw flash interface.
If we did decide it was an issue, OFW could simply write zeroes to the
unused areas, which would slow down the reflashing process but would not
be a showstopper to this implementation...

I am also assuming that (if we don't zero out the unused areas) that the
filesystems we use do not have the property of requiring that the whole
disk be zeroed out before the filesystem is created.. but I think this
is a safe assumption to make.

Another assumption is made about the size of the XO-1.5 disk. The sparse
file created in the very first dd command needs to be exactly the size
of the target partition (or at least no larger than it). Except for the
4GB/8GB choice, will all XO-1.5s coming off the production line have
exactly the same size disk? I guess we are in danger of this changing
slightly if we run into the requirement (or desire) of changing flash
chip manufacturer to one that has a slightly different idea of how big
4GB actually is.



Another option is to move a lot more filesystem logic into OFW. OFW
could implement an equivalent of mke2fs and hence could make ext2
filesystems on-demand. OFW could also grow the ability to create and
write files on such a filesystem.
We could then put all of the filesystem data as a tree in a .zip file
(or if .zip doesn't encode enough info, we could use .tar and
implement .tar file parsing in OFW) and OFW could create an ext2
filesystem of appropriate size and put the files in place.
However this approach would certainly require a lot more development
time, simply because it requires a lot of OFW code to be written, tested
and debugged.


Thoughts/ideas?
Daniel