Treatise on Formatting FLASH Storage Devices

[david at lang.hm]

On Wed, 4 Feb 2009, Mitch Bradley wrote:

> Benjamin M. Schwartz wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Mitch Bradley wrote:
>>
>>> It has been my experience that USB sticks and SD cards with intact
>>> factory formatting tend to last longer and run faster than ones that
>>> have been reformatted with random layouts.
>>>
>>
>> This gives us Linux users a bit of a dilemma if we want to use FTL flash
>> for primary storage.  FAT does not provide the file access permissions,
>> symlinks, hardlinks, or even case sensitivity, that we desire for most
>> filesystems on unixy systems.  However, FTL devices behave as a sort of
>> FAT-oriented black box, full of secret proprietary firmware that loves
>> FAT.
>
> I think that FTLs are getting better over time, so maybe the
> FAT-specific optimizations are starting to be replaced by more generic
> algorithms.  The rapidly growing market for FLASH-based storage is
> certainly attracting lots of development dollars.
>
> In the absence of FAT-specific optimizations, perhaps properly-aligned
> ext2 layouts will work well.
>
> Another solution is to choose high-quality devices.  I've had good
> results with some models from SanDisk and Transcend.  But sometimes it
> comes at a cost penalty - the really good SanDisk "Extreme III" SD cards
> cost 2.5x the going rate for commodity cards of the same capacity.  The
> good cards appear to be rather more tolerant of "abuse" than the El
> Cheapo's.  But even with the tough ones, I think its prudent to treat
> them gently.

the small devices are cheap enough, I'll bet a lot of people would be 
willing to chip in a few bucks if someone were to orginize a controlled 
test (or possibly one of the hardware websites can be prodded into doing a 
long enough and large enough test to have some valid statistics)

>>  One obvious proposal, therefore, would be to use FAT for storage,
>> but wrap it with a layer that implements all our favorite POSIX stuff.
>>

the problem with this is that your access won't follow the FAT pattern 
anymore, it will follow the pattern of the higher-level filesystem.

also, most setups like this that I have seen concentrate the 'extra' 
metadata for read efficiancy, but that's exactly the wrong thing to do for 
flash.

David Lang