Sample large datastore

[Tomeu Vizoso]

On Tue, May 20, 2008 at 3:11 AM, Martin Langhoff
<martin.langhoff at gmail.com> wrote:
> On Mon, May 19, 2008 at 9:26 PM, Tomeu Vizoso <tomeu at tomeuvizoso.net> wrote:
>> On Mon, May 19, 2008 at 10:30 AM, Tomeu Vizoso <tomeu at tomeuvizoso.net> wrote:
>>>
>>>    results, count = data_store.find({}, ['uid'])
>>
>> You may want to limit the amount of results returned by
>>
>> results, count = data_store.find({'limit': 20}, ['uid'])
>>
>> as trying to get too much info from the DS in one go can make things
>> very slow and ultimately OOM.
>
> Ah. Uhmmm... Does the "data object" in the datastore contain only the
> metadata? Or do we get a copy of the file stashed in a variable?

Just the metadata, but it will include the preview, so it can get
quite big as the results get prepared on the DS, sent over the wire,
received on the client side, and demarashalled there.

>  ... this is worrying... though at least it would explain in one
> simple blow the OOM and perf problems.

You weren't around yet when we used to index the base64 codification
of the preview :P

Regards,

Tomeu