[PATCH] Maintain a metadata copy outside the index (was Re: Datastore & backup - request for help)

[Martin Langhoff]

On Thu, May 22, 2008 at 6:14 AM, Tomeu Vizoso <tomeu at tomeuvizoso.net> wrote:
> the patch attached maintains a copy of the metadata of each object
> outside the xapian index. How it works:

Fantastic. Except that... erm... arhm... you forgot the patch ;-)

> - at every create and update, a json file is created next to the object's file,
>
> - it's also deleted along the object,
>
> - at startup, if the file <datastore_path>/.metadata.exported doesn't
> exist, check how many objects need to get their metadata exported
> (0.8s for 3000 entries)

That's pretty good.

> - in an idle callback, process each of those objects one per iteration
> (3ms per entry with simplejson, 2ms with cjson).

Exporting a few 100 per iteration probably is more efficient ;-)

> In my tests this has worked quite well, but I have one concern: can
> something bad happen if we have 20k files in the same dir (for a
> journal with 10k entries)?

Ok, we can split it into a subdir (which will only have 10K files then).

If there's a cost to large dirs in jfffs2 then we can use hashed dirs,
and that change will be needed for both the main datastore storage
_and_ the metadata files.

> One side effect of this is that when (if) we agree on a new on-disk
> data structure for the DS, it will be easier to convert than if we had
> to extract all the metadata from the index.

Yes. And as you said earlier, easy recovery if xapian goes to la-la land.

cheers,



m
-- 
 martin.langhoff at gmail.com
 martin at laptop.org -- School Server Architect
 - ask interesting questions
 - don't get distracted with shiny stuff - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff