XO-1.5 users: need your SD card data

[James Cameron]

On Mon, Apr 18, 2011 at 04:02:08PM +0100, Daniel Drake wrote:
> On 15 April 2011 03:10, James Cameron <quozl at laptop.org> wrote:
> > G'day Daniel,
> >
> > As an alternative, consider identifying the unused blocks in the
> > filesystem, and avoid including them in the .zd file. ?This would make
> > it unnecessary to know whether the bits will be set or cleared by the
> > card. ?ext2, ext3, and ext4 care nothing for the bits in unused blocks.
> 
> I'm not sure about that. I have been working under the assumption that
> ext* requires unused blocks to be zeroed out (or to read back 0s if
> read), which seems to be in agreement with the rest of your mail.

No, ext* does not require unused filesystem blocks to be zeroed out or
to read back 0s if read.

I don't see how you thought that would be in agreement with the rest of
my mail, which went on to describe how zerofree can be used to identify
unused filesystem blocks.  Once identified they may be omitted from the
.zd file.

mke2fs does not write zeroes to all unused filesystem blocks.  I've
tested this and the results are below.

A file deletion that causes filesystem blocks to become unused does not
write zeroes to the blocks.  While deleting a large file on ext3 can
take a while, it is not because it is zeroing the blocks.

> However, if we use a SD card that reads back all-ones for unused
> blocks, I think we would run into problems.

Assuming you are dealing with unused filesystem blocks ... I'm certain
you would not run into problems with an SD card that reads back all-ones
for erased blocks.  If you are running into problems, then perhaps they
relate to the failure of Q3A64 to properly erase the SD card during
fs-update, which I covered in another thread and since fixed.

As confirmation that reading back all-ones for unused filesystem blocks
is harmless, I've made a variant of zerofree that changes all the unused
filesystem blocks to 0xff, and the laptop operated normally after
reboot.  The patch for this is attached as zerofree-fill-0xff.patch

As confirmation that mke2fs does not write zeroes to all unused
filesystem blocks, a test method is to make a file containing non-zero
but identifiable data, make a filesystem over the file, count the number
of blocks that have been changed, and then use zerofree to demonstrate
that there are non-zero free blocks:

# create a psuedo-random 512-byte block
cd /tmp
dd if=/dev/urandom bs=512 count=1 of=key

# replicate it to 1MB size
rm -f key.1mb
for x in $(seq 2048); do cat key >> key.1mb; done

# replicate it to 4GB size
MB=4096
rm -f img
for x in $(seq ${MB}); do echo -ne "$x\r"; cat key.1mb >> img; done

# verify that all image blocks contain the original pattern
# source:
# http://dev.laptop.org/~quozl/count-changed-blocks.c (download)
# http://dev.laptop.org/~quozl/count-changed-blocks.c.html (view)
count-changed-blocks key img
# result: zero blocks changed from original pattern

# create an ext3 filesystem on the image,
# which will write filesystem metadata.
mke2fs -F -j img

# count the blocks changed by mke2fs
count-changed-blocks key img > mke2fs-changed-blocks
# result: 396294 times 512-byte blocks
# (equivalent to about 96 filesystem blocks, or 193MB)

# count and zero the free filesystem blocks
zerofree -v img
# result: 997249/997265/1048576 nonzero/free/total 4096-byte blocks

If mke2fs did write zeroes to unused filesystem blocks, then (a) the
number of changed image blocks would have approached the size of the
image, and (b) zerofree would have reported no filesystem blocks
nonzero.

olpc-os-builder does start with an image file that is derived from
/dev/zero, per modules/sd_card_image/image.50.makefs.sh ... therefore
unused filesystem blocks will typically be zeroed in the .zd file, but
the converse is not true: a zeroed image block in the .zd file won't
necessarily be an unused filesystem block.

What I'm proposing is that unused filesystem blocks not be embedded in
the .zd file, and as a result fs-update will erase the corresponding SD
block but not write to it.  Since writing to the SD card is the most
expensive operation, time will be saved, and it won't be necessary to
know what the card sets the bits to on erase.

(For reference, on the Toshiba SD parts I've been testing, an
erase-blocks operation is very fast, and apparently only changes
metadata in the card's flash translation layer ... but doesn't actually
perform flash level erasure.  Physical erasure still has to occur during
write.  The fs-update elapsed time is unchanged.)

-- 
James Cameron
http://quozl.linux.org.au/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: zerofree-fill-0xff.patch
Type: text/x-diff
Size: 645 bytes
Desc: not available
URL: <http://lists.laptop.org/pipermail/devel/attachments/20110419/e65c0c5b/attachment.patch>