[PATCH] Maintain a metadata copy outside the index (was Re: Datastore & backup - request for help)

[Tomeu Vizoso]

On Thu, May 22, 2008 at 5:48 AM, Benjamin M. Schwartz
<bmschwar at fas.harvard.edu> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Jameson "Chema" Quinn wrote:
> |>> - at startup, if the file <datastore_path>/.metadata.exported doesn't
> |>> exist, check how many objects need to get their metadata exported
> |>> (0.8s for 3000 entries)
> |> That's pretty good.
> |>
> |>> - in an idle callback, process each of those objects one per iteration
> |>> (3ms per entry with simplejson, 2ms with cjson).
> |> Exporting a few 100 per iteration probably is more efficient ;-)
> |
> |
> | This brings up the issue of TamTam imperfect timing - it would be great if
> | there were some way to turn off all unnecessary background CPU use for cases
> | like TamTam. If so, I'd say 12*3ms is about the right size for a background
> | click every second or two.
> |
>
> We are free to set nice values for each process as appropriate.  Backup
> should obviously get a very high nice value.

This process is easier to perform inside the DS process and will
happen only on first boot after update. But the backup process itself
(rsync) should have a really high nice value, agreed.

Thanks,

Tomeu