Treatise on Formatting FLASH Storage Devices

[Mitch Bradley]

Benjamin M. Schwartz wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Mitch Bradley wrote:
>   
>> It has been my experience that USB sticks and SD cards with intact 
>> factory formatting tend to last longer and run faster than ones that 
>> have been reformatted with random layouts.
>>     
>
> This gives us Linux users a bit of a dilemma if we want to use FTL flash
> for primary storage.  FAT does not provide the file access permissions,
> symlinks, hardlinks, or even case sensitivity, that we desire for most
> filesystems on unixy systems.  However, FTL devices behave as a sort of
> FAT-oriented black box, full of secret proprietary firmware that loves
> FAT. 

I think that FTLs are getting better over time, so maybe the 
FAT-specific optimizations are starting to be replaced by more generic 
algorithms.  The rapidly growing market for FLASH-based storage is 
certainly attracting lots of development dollars.

In the absence of FAT-specific optimizations, perhaps properly-aligned 
ext2 layouts will work well.

Another solution is to choose high-quality devices.  I've had good 
results with some models from SanDisk and Transcend.  But sometimes it 
comes at a cost penalty - the really good SanDisk "Extreme III" SD cards 
cost 2.5x the going rate for commodity cards of the same capacity.  The 
good cards appear to be rather more tolerant of "abuse" than the El 
Cheapo's.  But even with the tough ones, I think its prudent to treat 
them gently.

>  One obvious proposal, therefore, would be to use FAT for storage,
> but wrap it with a layer that implements all our favorite POSIX stuff.
>   

Puppy Linux does something like that, using a (FAT, ISO9660, or 
whatever) file as a container for an ext2 filesystem image.

The practice is also rather common in the world of virtualization.  My 
primary Linux system is actually a colinux "vm" running under Windows 
with the ext2 filesystem image inside an NTFS file (actually three 
files, one for root, one for home, and one for miscellaneous big wads of 
client-specific stuff) .  That's proven itself very convenient over 
time; I've transported and mixed-and-matched those filesystem images to 
several different host machines.  Based on that and other pleasant 
experiences with VM filesystem "snapshots", I'm of the opinion that a 
quiet revolution is brewing in the way that people deal with filesystems 
inside files.

You could treat an existing FAT filesystem as a very flexible 
partitioning scheme.  You can make any number of partitions, of any 
size, with a hierarchical namespace, with a good collection of tools for 
manipulating them.


> This has been done before for Linux, in the guise of UMSDOS/UVFAT [1][2].
>  Although that work has fallen out of date, I suspect one could
> reimplement it quickly using new linux features such as FUSE.  The
> question is: would such an approach be worthwhile?
>   

I think that the wrapped-metadata approach has largely run its course.  
There used to be a lot of activity in that area, but I haven't seen much 
of interest lately.  I think that containerization is likely to be the 
happening thing for the near term.

> - --Ben
>
> [1] http://linux.voyager.hr/umsdos/
> [2] http://en.wikipedia.org/wiki/UMSDOS
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.9 (GNU/Linux)
>
> iEYEARECAAYFAkmKBJ0ACgkQUJT6e6HFtqR8kwCfc9MlcbGv1yaSEog6lNJoqmey
> kE0AmwRxwXtORZSITzyDUW5gqu9xBpoq
> =Kxa1
> -----END PGP SIGNATURE-----
>