zhashfs: write first block last

[James Cameron]

On Thu, May 05, 2011 at 08:08:09PM +0100, Daniel Drake wrote:
> Is it possible to fix the erasing?

Yes, but it will take many seconds to do it slowly enough for the few
cards that are not happy about doing it quickly.  This will slow
fs-update still further.

(We might invest more in guessing how the cards expect to be treated for
this CMD38, but having spent a while on it already I fear it will be
another long while before we get there.)

We might instead ensure that block zero is erased, by writing a zero
block.  That might be implemented either in fs-update or in zhashfs.  I
prefer it be implemented in zhashfs as an option, so that fs-update is
most flexible.

> Perhaps nandblaster could do it once it has confirmed that there's a
> server transmitting, before starting the receive process. It has a few
> advantages:

A disadvantage is that an interrupted NANDblaster receive would then
result in a laptop with no operating system.  At the moment, an
interrupted receive does not break things.

An erase might instead be done just prior to the writing of the .zd
file to the SD card.  (I'm still not in favour of an erase).

> - nicely solves this issue :)
> 
> - Second-guessing how cards actually work, I expect having less SD
> card sectors occupied means less work for the housekeeping routines of
> the embedded flash controller

None of the cards I tested (from XO-1.5 B1, B2, C1, C2, C3, plus others)
showed any increase in general performance after being fully erased
using the CMD38 method.  I had tested boot time, and journal entry
creation.

Based on this I guess the cards work by updating pointers.

> - Guesswork again, writing to erased sectors is probably quicker than
> writing to occupied sectors. Flash memory needs erase-before-write,
> right? Hopefully not if its already erased.

None of the cards I tested showed any increase in sequential write
performance after being fully erased using the CMD38 method.  I had
tested fs-update elapsed time.  I had hoped it would be faster, but it
was not.

Based on this I guess the cards work by not erasing the flash memory,
but instead relying on updating pointers.

(The next card we try might be different, but we are choosing cheap
cards because low cost is a goal.)

> - I'm hoping that moving to lazy inode table initialization means that
> we can implement quick on-the-fly resizing to utilise all SD card
> space in future. The one danger with this is block group descriptors
> being reused from a previous filesystem which has some remnants left
> on disk:

Given that this scenario only occurs if there is corruption to the block
group descriptors ... a scenario which is more than likely going to
cause unpredictable results anyway ... it is far more likely that a
laptop won't boot and will be reinstalled.

> - Around 4% of the writes of a sparse zd file are zero-blocks. I'm
> still hoping to persuade someone to implement the logic I originally
> proposed (in addition to the sparse thing) to further speed up
> fs-update in the common case. The disk needs to be pre-erased for that
> to work.

But it will only work on cards that erase to zero.

Another gain might be had by reading the block and checking for match
before writing it.  Reads are much faster than writes.  Since a read of
a 4GB card costs four minutes, and a typical .zd is 50% sparse, this
would add two minutes to fs-update, and either;

- subtract seven minutes if the fs-update is a reinstallation, (total
  fs-update time two minutes),

- subtract, say, six minutes if the fs-update was an upgrade of a
  deployment master image, (total fs-update time three minutes),

- have no effect on a major version upgrade, since no attempt is made to
  keep common files in the same place on the image, (total fs-update
  time nine minutes),

This gain would require that erase not be done.

> - It's nice to have for the privacy aspect

I'd prefer that we publish a privacy guide that explains how to erase
the storage manually.

I think we should focus on reducing the cost of deployment ... in
particular the time taken to install a deployment image.

I've another idea for that which I tried to explain to Martin the other
day ... a .zd file describing the delta from the as-shipped
manufacturing image, and the deployment image.  A deployment team would
augment the as-shipped image, generate a delta, and install it quickly
on laptops as they are unboxed.  Total fs-update time would be a few
seconds, depending on the degree of change.  Laptops that come out of
the box with the wrong image would trigger an exception for special
treatment.  fs-update must not erase for this to work.

-- 
James Cameron
http://quozl.linux.org.au/