[Server-devel] [IIAB] [XSCE] RE: [UKids] Re: [support-gang] Taking OpenStreetMap Offline - DESIGN Call - Thur June 11, 10AM EDT / 2PM UTC

Sat Jun 20 22:14:17 EDT 2015

Hi all

I wanted to do some testing of the search solutions on some kind of
hardware/software that would look similar to what will be deployed in the
field (to measure the performance impact).

Got this idea from this post where :
http://www.openstreetmap.org/user/SimonPoole/diary/34857

Should I be testing with XSCE 5.1? XSCE 5.5 beta? Should I get some
hardware like a Pi2, or just get some VM? Any VM recommendation?

Thanks

On Wed, Jun 17, 2015 at 3:15 PM, Tim Moody <tim at timmoody.com> wrote:

> Thanks, this is very useful information.  I didn't realize that slippy
> maps is a strategy and leaflet is a library for it that OSM itself uses.
>
>
>
> A few questions.
>
>
>
> Which of the filtered datasets did you use for populated places?
>
>
>
> Are they from
> http://wiki.openstreetmap.org/wiki/Planet.osm#Country_and_area_extracts ?
>
>
>
> Is the directory structure such that separately rendered tiles for
> different regions can be merged by over-copying?
>
>
>
> I did some sampling.
>
>
>
> In Manhattan the maximum zoom is
> http://xsce/iiab/maps/tile/15/9649/12318.png so level 15 (16th)
>
>
>
> In Northern Ontario the maximum zoom is
> http://xsce/iiab/maps/tile/13/2048/2704.png level 13 (14th)
>
>
>
> In the middle of the North Pacific
> http://xsce/iiab/maps/tile/14/1088/6190.png a surprising level 14
>
>
>
> Moving slightly left I get http://xsce/iiab/maps/tile/14/1080/6191.png so
> I'm not quite clear on the generic blue tile point.
>
>
>
> I'm also not clear on the point about the meta tiles.  From what I read in
> http://wiki.openstreetmap.org/wiki/Meta_tiles , meta files just
> concatenate the smaller tiles, so I'm not clear on how that saves space.  I
> do understand that it greatly reduces the number of nodes in the file
> system making retrieving them viable.
>
>
>
> I looked at space usage by zoom level on the current IIAB OSM:
>
>
>
> Level
>
> Size (in K)
>
> Cumulative (in K)
>
> Cumulative (in G)
>
> 0
>
> 60
>
> 60
>
> 0
>
> 1
>
> 84
>
> 144
>
> 0
>
> 2
>
> 152
>
> 296
>
> 0
>
> 3
>
> 300
>
> 596
>
> 0
>
> 4
>
> 1,004
>
> 1,600
>
> 0
>
> 5
>
> 2,800
>
> 4,400
>
> 0
>
> 6
>
> 9,064
>
> 13,464
>
> 0
>
> 7
>
> 26,328
>
> 39,792
>
> 0
>
> 8
>
> 86,280
>
> 126,072
>
> 0
>
> 9
>
> 255,576
>
> 381,648
>
> 0
>
> 10
>
> 859,540
>
> 1,241,188
>
> 1
>
> 11
>
> 2,233,136
>
> 3,474,324
>
> 3
>
> 12
>
> 5,905,172
>
> 9,379,496
>
> 9
>
> 13
>
> 17,480,260
>
> 26,859,756
>
> 27
>
> 14
>
> 43,947,380
>
> 70,807,136
>
> 71
>
> 15
>
> 35,454,488
>
> 106,261,624
>
> 106
>
>
>
> So 10G gets 13 levels.
>
>
>
>
>
> *From:* unleashkids at googlegroups.com [mailto:unleashkids at googlegroups.com]
> *On Behalf Of *Braddock Gaskill
> *Sent:* Monday, June 15, 2015 8:39 PM
> *To:* Internet In a Box Working Group
> *Cc:* xsce-devel; Unleash Kids!; Jaakko Helleranta; server-devel; iaep;
> Community Support Volunteers -- who help respond to help AT laptop.org
> *Subject:* Re: [IIAB] [XSCE] RE: [UKids] Re: [support-gang] Taking
> OpenStreetMap Offline - DESIGN Call - Thur June 11, 10AM EDT / 2PM UTC
>
>
>
> In IIAB we use a dataset of populated places to render at the highest
> resolution level, and render the rest of the world at a slightly lower
> resolution.  Our highest resolution is high enough to see individual
> buildings.  Our total dataset of rendered tiles for the world with detailed
> populated places coverage is 100GB.
>
>
>
> Another optimization we made for IIAB that contributed considerable
> dataset size reduction and helped with filesystem efficiency - we use the
> OSM "meta" tiles, which normally store 64 tile png images, but we modify
> them to store 4096 png images per file, and we made an optimization so that
> identical tiles are removed (think of all the blue of the ocean cover 75%
> of the earth).
>
>
>
> We use Leaflet as our javascript web front end, and it works quite well.
>
>
>
> -braddock
>
>
>
>
>
> On Sun, Jun 14, 2015 at 11:24 PM, Nick Doiron <ndoiron at mapmeld.com> wrote:
>
> I don't think there's any technical issues with rendering the world at 10
> and specific countries at 16, other than the human knowing where they can
> and cannot zoom
>
>
>
> -- Nick
>
>
>
> On Sun, Jun 14, 2015 at 11:58 AM, Tim Moody <tim at timmoody.com> wrote:
>
> couple of observations:
>
>
>
> As expected, the new tiles have a lot more detail.
>
>
>
> There are more levels of zoom in the new ones.
>
>
>
> Some names have changed - the old map had Lalitpur and the new one has
> Patan (both are used)
>
>
>
> I don't see any boxes for unprintable characters, but there is a lot less
> Devanagari. (Google maps has more)
>
>
>
> Is it possible to merge individually generated regional tiles?  for
> example if you rendered India and Nepal separately would you get both?
>
>
>
> What happens if you render the world at level 10 and then specific
> countries at 16?
>
>
>
> *From:* xsce-devel at googlegroups.com [mailto:xsce-devel at googlegroups.com] *On
> Behalf Of *Anish Mangal
> *Sent:* Saturday, June 13, 2015 10:06 PM
> *To:* Jérôme Gagnon-Voyer
> *Cc:* xsce-devel; Community Support Volunteers -- who help respond to
> help AT laptop.org; Unleash Kids!; server-devel; iaep; Internet In a Box
> Working Group; Jaakko Helleranta
> *Subject:* Re: [XSCE] RE: [UKids] Re: [support-gang] Taking OpenStreetMap
> Offline - DESIGN Call - Thur June 11, 10AM EDT / 2PM UTC
>
>
>
> Hi,
>
> Following from the skype call this week, I uploaded Nepal's OSM data in
> postgres and render tiles to see performance, and as expected, everything
> is blazing fast.
>
> The pbf file went into the postgress in less than 5 minutes, and tiles are
> being rendered pretty fast as well.
>
> You can check it yourself by going to
> http://home.braddock.com:28112/osm/slippymap.html
>
> * Select Mapnik
>
> * Zoom out and center over Nepal
>
> * Select Local tiles
>
> * Zoom in
>
> I don't know what this "proves" as the bottleneck still is doing this for
> the entire planet, which we need to figure out a way for anyway.
>
> Best,
>
> Anish
>
>
>
>
>
> On Sun, Jun 14, 2015 at 6:02 AM, Anish Mangal <anishmg at umich.edu> wrote:
>
> Hi Jérôme,
>
> I incorporated one of your comments; as for the other I think Timm, Nick
> would be better suited for the discussion (python backend v/s frontend js).
> :-)
>
> Best,
>
> Anish
>
>
>
> On Sun, Jun 14, 2015 at 3:22 AM, Jérôme Gagnon-Voyer <gagnonje at gmail.com>
> wrote:
>
> Hi Anish
>
>
>
> Great document. I've added 2 comments to the document, feel free to
> incorporate into the document if that makes sense.
>
>
>
> Also added a TODO for myself to do more research about the various search
> solutions.
>
> I've used Nominatim in the past (on the client side, not the server
> infrastructure) and it was overall very good, but I'll want to know more
> about the other ones.
>
>
>
> On Sat, Jun 13, 2015 at 12:51 PM, Tim Moody <tim at timmoody.com> wrote:
>
> Thanks, Anish for an excellent start at getting this down on paper (so to
> speak).  I think this covers things pretty well and gives us the necessary
> hooks on which to hang the details as we begin fleshing out solutions to
> the requirements you documented.
>
>
>
> Under issues I added one point about rendering non-Roman character sets.
>
>
>
> *From:* unleashkids at googlegroups.com [mailto:unleashkids at googlegroups.com]
> *On Behalf Of *Anish Mangal
> *Sent:* Saturday, June 13, 2015 12:15 PM
> *To:* Community Support Volunteers -- who help respond to help AT
> laptop.org
> *Cc:* Unleash Kids!; server-devel; xsce-devel; iaep; Internet In a Box
> Working Group; Jaakko Helleranta
> *Subject:* [UKids] Re: [support-gang] Taking OpenStreetMap Offline -
> DESIGN Call - Thur June 11, 10AM EDT / 2PM UTC
>
>
>
> Okay, so I tried to encapsulate whatever we discussed into a design
> document, which can hopefully serve as a base for reaching out to the OSM
> community and to better organize our own thoughts.
>
>
> https://docs.google.com/document/d/1LWsN-cPH3lvMuXS-f0Tk8IWVh-3X808WoIt0OJ-QNt8/edit#
> <https://docs.google.com/document/d/1LWsN-cPH3lvMuXS-f0Tk8IWVh-3X808WoIt0OJ-QNt8/edit>
>
> Please feel free to edit the information there as I may have missed some
> points, or interpreted them differently than as intended.
>
> Best,
>
> Anish
>
>
>
> On Fri, Jun 5, 2015 at 12:00 AM, Adam Holt <holt at laptop.org> wrote:
>
> Who's attending http://stateofthemap.us at the UN in NYC this weekend?
>
> Who's most serious about bringing OpenStreetMap's opportunities and rapid
> progress into the hands of the world's offline poor -- kids and all,
> American and Swahili?  Wanna Take This Map Outside, offline and off the
> grid, where we all belong/began?
>
> http://Internet-in-a-Box.org and similar efforts have made a gigantic 1st
> step: in Ghana and Rwanda I could never have imagined better reviews to our
> 2015 deployments begun there in recent months.  Many are now changing the
> game increasingly for the Bottom BillionS, among many who've literally
> never seen a globe before, nevermind a map of their own towns.  Both in
> OLPC (school) contexts, in libraries, on Nepali hillsides where folks don't
> have proper homes, and far beyond --- perspectives (literally) are about to
> change.
>
> But modern phones today contain more than enough gigabytes to display ALL
> OpenStreetMap map detail within most countries, and yet they do not yet,
> WHY?  Where are the Khan Academies and KA Lites of OSM (OpenStreetMap)
> bringing new classroom/journalistic rubrics, freeing everyone's "brain
> software" to explore and document our own communities in our own languages?
>
> *Who Will Take The Next Steps?*
> *What engineering middleware, distribution vectors, community/economic
> models, and final field packagings will get us all there & beyond?*
>
> *Will offline edit-contributions prove impossible, much like with
> Wikipedia in Peru, when offline kids edit overly stale
> OpenStreetMap/Wikipedia images a year before/later?*
>
>
>
> As such the wider OLPC community is hosting a DESIGN Call to bring forward
> ideas across the OpenStreetMap landscape, fertilizing our immediate work
> with school server projects like http://schoolserver.org, http://xsce.org,
> https://en.opensuse.org/openSUSE:Education-Li-f-e ETC.  Ministries of
> Education in India elsewhere are watching closely, expressly eager to help
> if we can point the way.  All giving our "2020 Vision" questions very
> practical and immediate "customers" well before 2020, much like Garmin GPS
> units fed a wonderful ecosystem of geo-specific "gmapsupp.img" offline map
> files over the past decade:
> http://wiki.openstreetmap.org/wiki/OSM_Map_On_Garmin/Download
>
> That decade's now done: what IS our framework for the coming decade?
> Please join us if you have strategic/partnership ideas towards making
> Offline OSM Designs happen, *Thank You !!*
>
>
>
> 10AM New York Time / 2PM UTC
>
> Thursday, June 11th
>
> RSVP with your Skype username, if we are more than 15-20 we'll use an
> industrial conference call system instead!
>
>
> CONCLUSION: Could OpenStreetMap be the very ultimate in Constructionist
> Learning Projects, replacing OLPC in coming years, on a quickly
> shrinking/endangered but still green-in-parts planet?  Regardless, how to
> build the OLPC Movement's offline/civic learning successes, consciously
> learning from its community infrastructural mistakes?  Nick Doiron (
> http://mapmeld.com) and Anish Mangal (
> https://in.linkedin.com/in/anishmangal) who've spent years thinking about
> this topic will lead the discussion, agenda is entirely yours!  OPTIONAL:
> submit agenda items in advance right here:
>
>
> https://docs.google.com/document/d/1o6QtzLb6e58YKWqMf_junux2XyBRLFm31un8YLcYslg
>
>
>
> *< forward invitation to Twitter/wherever as appropriate >*
>
>
> --
>
> Unsung Heroes of OLPC, interviewed live @ http://unleashkids.org !
>
>
> _______________________________________________
> support-gang mailing list
> support-gang at lists.laptop.org
> http://lists.laptop.org/listinfo/support-gang
>
>
>
> --
> Unsung Heroes of OLPC, interviewed live @ http://unleashkids.org !
> ---
> You received this message because you are subscribed to the Google Groups
> "Unleash Kids" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to unleashkids+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
>
> --
>
> Jérôme Gagnon-Voyer
>
>
>
> --
>
>
>
>
>
>
>
>
>
>
>
> --
> Unsung Heroes of OLPC, interviewed live @ http://unleashkids.org !
> ---
> You received this message because you are subscribed to the Google Groups
> "Unleash Kids" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to unleashkids+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
Jérôme Gagnon-Voyer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.laptop.org/pipermail/server-devel/attachments/20150620/ec142044/attachment-0001.html>