json parser in python - stream-based?

[Martin Langhoff]

On Wed, May 20, 2009 at 12:33 PM, Tomeu Vizoso <tomeu at sugarlabs.org> wrote:
> Cannot directly answer your questions, but my guess is that json is
> expected to be used in small files. Any chance to split that big file
> in several smaller ones?

Yes, but it changes one problem with lots of other problems.

For large deployments, you want to avoid opening *all* the files --
any file / directory hashing scheme needs to be portable across
filesystems (mainly to those of the FAT variety). Plus user training.

A targz or cpio may be viable for a single file solution that hides
many files inside. But overall, if we're changing formats, a
line-oriented format will do. Python can read a huge line-oriented
file quickly without choking.

Either way, if we change the format, it won't be interoperable with
older builds. And the user documentation will be pure pain: -- between
build versions and filesize, you end up with 4 cases. Imagine the
documentation :-/

I want to "just fix the problem" without causing ripples elsewhere in
the environs or complicating people's lives. A single human-readable
file is a perfectly good vehicle, and we have that already. And a
modest programming effort should be able to solve it... :-)

It'll be much better -- and generalizable -- if the kind of lib I'm
thinking of exists.

cheers,


m
-- 
 martin.langhoff at gmail.com
 martin at laptop.org -- School Server Architect
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff