[Server-devel] [UKids] Re: [XSCE] Tilestache on XSCE stack (OSM offline)

Tue Aug 11 10:18:54 EDT 2015

Hey Anish

Not sure if you read what I wrote in the design document for the
architecture, but the idea was to have it cloud based. Generate the tiles
in EC2, and then host it in S3. Of course we would eventually need to
estimate how much that would cost.

Not sure yet if that's the best solution but that's what I'm proposing

We also have to keep in mind that if we generate the tiles on the hardware
that you mentioned, we still need to transfer the output somewhere to host
it (and pay for the bandwidth fee to upload at that point).
If we generate on EC2, we pay for the usage of EC2, but the transfer of the
output to S3 is entirely free.

My initial idea was to have something that is more "maintainable" in the
cloud, where the entire tile generation can be re-trigerred when you desire
it and something that can easily be changed/updated based on your need. I'm
just afraid that if it's a private server, this might not be there in the
long term, and not necessarily easy to give access to it. Just trying to
come up with a solution that is not too "manual" (ie: moving files around
through SSH through the server of someone else that only few people
understand), which eventually means the tiles don't get updated often and
might stay stale for a long term.

Of course, open to other ideas, just my proposition here.

On Tue, Aug 11, 2015 at 9:35 AM, Anish Mangal <anishmg at umich.edu> wrote:

>
> Hi
>
> Here is the data
> size  zoom
>  28K     0
>  48K     1
>  80K     2
> 184K     3
> 340K     4
> 788K     5
> 2.1M     6
> 5.8M     7
>  19M     8
>  58M     9
> 188M    10
> 673M    11
>  17M    12
>  54M    13
> 134M    14
> 375M    15
> 1.1G    16
> 3.0G    17
> 9.6G    18
>
> As is obvious, for zoom levels 0-11 I had generated tiles for the entire
> world, which was essentially empty tiles except the shape of continents
> since the only data source I used was the nepal osm data
>
> Zoom level 12 and onwards, tiles were generated only for nepal.
>
> The bigger question i'd be asking is the setup required to generate tiles,
> and what it would take to do for the entire planet. I guess Braddock would
> be happy to give access to Jerome on hadron if he wants to give it a try.
> It is a machine with a 500G SSD and 32G RAM and an 8 core processor.
>
>
>
> On Tue, Aug 11, 2015 at 6:38 PM, Tim Moody <tim at timmoody.com> wrote:
>
>> Anish, you generated tiles for Nepal.  How does the size compare to what
>> you got?
>>
>>
>>
>> *From:* unleashkids at googlegroups.com [mailto:unleashkids at googlegroups.com]
>> *On Behalf Of *Tim Moody
>> *Sent:* Tuesday, August 11, 2015 9:04 AM
>> *To:* 'Jérôme Gagnon-Voyer' <gagnonje at gmail.com>
>> *Cc:* 'Adam Holt' <holt at laptop.org>; 'xsce-devel' <
>> xsce-devel at googlegroups.com>; 'Unleash Kids!' <
>> unleashkids at googlegroups.com>; 'server-devel' <
>> server-devel at lists.laptop.org>
>> *Subject:* RE: [UKids] Re: [XSCE] Tilestache on XSCE stack (OSM offline)
>>
>>
>>
>> Can I get the data, maybe a dropbox link?  not sure of a place to ftp
>> unless you have one.
>>
>>
>>
>> *From:* unleashkids at googlegroups.com [mailto:unleashkids at googlegroups.com
>> <unleashkids at googlegroups.com>] *On Behalf Of *Jérôme Gagnon-Voyer
>> *Sent:* Tuesday, August 11, 2015 9:01 AM
>> *To:* Tim Moody <tim at timmoody.com>
>> *Cc:* Adam Holt <holt at laptop.org>; xsce-devel <
>> xsce-devel at googlegroups.com>; Unleash Kids! <unleashkids at googlegroups.com>;
>> server-devel <server-devel at lists.laptop.org>
>> *Subject:* Re: [UKids] Re: [XSCE] Tilestache on XSCE stack (OSM offline)
>>
>>
>>
>> You define the path to the mbtiles in tilestache.cfg, so yes, they could
>> be in any location (as long as the access rights / permissions are right)
>>
>>
>>
>> On Tue, Aug 11, 2015 at 8:28 AM, Tim Moody <tim at timmoody.com> wrote:
>>
>> This looks great and not overly complex.  XSCE uses one convention that
>> you are probably not aware of and that is that we put data in /library, in
>> this case probably /library/osm or tilestache or some combination.  Can
>> tilestache.cfg point to a location different than the html and config files?
>>
>>
>>
>> *From:* unleashkids at googlegroups.com [mailto:unleashkids at googlegroups.com]
>> *On Behalf Of *Adam Holt
>> *Sent:* Tuesday, August 11, 2015 12:19 AM
>> *To:* xsce-devel <xsce-devel at googlegroups.com>; Unleash Kids! <
>> unleashkids at googlegroups.com>; server-devel <
>> server-devel at lists.laptop.org>
>> *Subject:* [UKids] Re: [XSCE] Tilestache on XSCE stack (OSM offline)
>>
>>
>>
>> Don't stop pushing Jerome- I hope you can demonstrate Nepal-or-larger
>> working models during our Toronto summit Oct 10-12.
>>
>> See you 7AM EDT Thursday on our weekly call, if you have time for our
>> unusually early meeting this week: http://tinyurl.com/xsceminutes
>>
>>
>> (Several of our early design discussions on offline mapping began on
>> unleashkids at googlegroups.com, so am bringing that thread full circle.)
>>
>>
>>
>> On Mon, Aug 10, 2015 at 11:01 PM, Anish Mangal <anishmg at umich.edu> wrote:
>>
>> adding server-devel (the publicly archived mailing list) as well.
>>
>> Great work. Thanks!
>>
>> Happy to help with the testing. I have an intel NUC running an atom
>> processor 2 Ghz, and 2GB RAM - not exactly low end, but still a viable
>> appliance to deploy the xsce.
>>
>> Could you tell me how to get the mbtiles from your ftp. If this works, I
>> can help with creating a playbook - and possibly admin console integration.
>>
>> Thanks,
>>
>> Anish
>>
>>
>>
>>
>>
>> On Tue, Aug 11, 2015 at 8:19 AM, Jérôme Gagnon-Voyer <gagnonje at gmail.com>
>> wrote:
>>
>> Hi all
>>
>>
>>
>> This is about the project to bring a new way of storing OpenStreetMaps
>> offline for internet in a box (schoolserver).
>>
>>
>>
>> (Apologies if I'm using the wrong mailing list, feel free to add more)
>>
>>
>>
>> I wanted to test what it would look like to run Tilestache to serve
>> .mbtiles using the schoolserver. So I installed XSCE on a local VM with
>> Fedora 21 here, installed Tilestache, moved the .mbtiles in it, and did
>> some testing.
>>
>> So far I only tested with Nepal zoom 0-13 and Nepal zoom 14-15 (to see
>> how combining multiple files was working)
>>
>> I reduced the VM to use only 1 core at 50% (I'm on MBP Intel i5 2.6Ghz, 2
>> cores). Speed was decent when viewing with leaflet in my browser from my
>> laptop.
>>
>>
>>
>> I've created a gist that explains all the steps required to do it
>>
>> https://gist.github.com/jeromegv/cb64bc2c9777794e80af
>>
>>
>>
>> The reason I wanted to test it was also to see if the integration would
>> be relatively straightforward. I assume you would eventually want to do the
>> install through the schoolserver admin console. When comes the time to do
>> that, I might not be the person to do this integration, but I think that
>> the steps I've done so far should at least prove that the XSCE stack should
>> be compatible.
>>
>>
>>
>>    - I might not have the best testing setup or what is similar to a
>>    typical schoolserver, if you have the proper hardware to test, let me know,
>>    I can send you the .mbtiles on a FTP for nepal and you can test it (about
>>    1Gb)
>>    - Feel free to provide feedback on the GIST and if the integration
>>    seems to make sense
>>    - There's an extra optimization that is possible for tilestache, to
>>    cache the tiles to the HD once they were retrieved and served from the
>>    .mbtiles once. It's currently disabled. Not so sure if it's worth it and
>>    how much we would gain considering MBtiles are already pre-rendered files,
>>    perhaps better to test once we have a full planet. The downside is that it
>>    stores on HD a lot of PNG files and this might eventually baloon if you get
>>    tons of people to use those maps. Benefit of sticking entirely to mbtiles
>>    is we know exactly how much disk space. Might be optional setting from
>>    admin console when there's plenty of HD available.
>>    - Next step for me will be to test generation of MBtiles for bigger
>>    country and eventually get to the Planet. Will need some type of cloud
>>    setup to manage/store all that.
>>
>> Thanks
>>
>>
>>
>> --
>>
>> Jérôme Gagnon-Voyer
>>
>> --
>>
>> Unsung Heroes of OLPC, interviewed live @ http://unleashkids.org !
>>
>> --
>> Unsung Heroes of OLPC, interviewed live @ http://unleashkids.org !
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Unleash Kids" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to unleashkids+unsubscribe at googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> Unsung Heroes of OLPC, interviewed live @ http://unleashkids.org !
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Unleash Kids" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to unleashkids+unsubscribe at googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>>
>>
>> --
>>
>> Jérôme Gagnon-Voyer
>>
>> --
>> Unsung Heroes of OLPC, interviewed live @ http://unleashkids.org !
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Unleash Kids" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to unleashkids+unsubscribe at googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> Unsung Heroes of OLPC, interviewed live @ http://unleashkids.org !
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Unleash Kids" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to unleashkids+unsubscribe at googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
Jérôme Gagnon-Voyer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.laptop.org/pipermail/server-devel/attachments/20150811/a7498c69/attachment-0001.html>