[Server-devel] [XSCE] sugarstats and xovis

[Sameer Verma]

On Thu, Jul 3, 2014 at 2:26 AM, Martin Dluhos <martin at gnu.org> wrote:
> On 06/27/2014 02:10 PM, Martin Abente wrote:
>> Hello Martin!
>>
>> Do you have documentation on how XOvis pipeline works? I would like to see how
>> it works after the data is store and the visualization is generated.
>
>
> Here is a simple graphic that illustrates the workflow:
>
> http://i.imgur.com/Xhy1Vx3.jpg
>
> XOvis is a Couch application, which means that it's also stored in the db along
> with the journal metadata. It queries the db using map-reduce queries, which you
> can see here:
>
> https://github.com/martasd/xovis/blob/master/lib/views.js
>
>> Writing a tool to export the metadata from Harvest-server to a compatible subset
>> in the visualization-ready format of XOvis, would be cheaper in development
>> cost, but more expensive in running time (with large datasets). The other way
>> around, would be to allow XOvis read directly from Harvest-server MySQL format.
>

CouchDB has two distinct advantages for XOVis. 1) It allows for
"eventual consistency" which helps in syncing data from the school to
a central location using XS or XSCE. I believe this isn't relevant for
AU/Harvest because they don't use a school server, and the XOs send
metadata directly. 2) CouchDB does the aggregation through mapreduce.
This is powerful, and withour couchDB (or mongodb etc) you'll have to
write the aggregation yourself and/or get MySQL to do it.

The visualization may or may not be stored in the DB. It just happens
to be one more convenience of CouchDB.

cheers,
Sameer

> Since transferring metadata into the db needs to be done only once, running time
> has never been a major concern. How large are the datasets you've collected on
> the Harvest-server?
>
> Cheers,
> Martin
>
>