[Sugar-devel] Killing activities when memory gets short

[Tomeu Vizoso]

2010/8/11 NoiseEHC <NoiseEHC at freemail.hu>:
>
>> We used to do that, the problem is that we don't control our platform
>> as Google controls Android and you need to make sure that resources
>> that need to be specific of each child process aren't shared (dbus and
>> X connections, etc).
>>
>> I'm personally more interested in reducing the amount of resources we
>> need to start activities (which is quite insane right now), but
>> sharing more of those resources sounds like a good idea to me.
>>
>
> Since I spent quite a bit of time analyzing the Python runtime here is my
> conclusion, maybe it will make wasting all that time less painful.
>
> First, most of the memory consumed by an activity is the process heap. While
> the Python runtime and native libraries are shared among processes, the heap
> is not. Even if you fork processes it will not work because reference
> counting will dirty the shared pages

This would hold true if a significant part of the stuff loaded during
startup was actually ever referenced. This is not the case in Sugar,
so you indeed can get quite some savings from the prefork trick.

But of course, the right fix here is not sharing that memory, but not
require it in the first place if we don't need it.

One example is to stop compiling regular expressions eagerly, as
mentioned in http://www.mail-archive.com/sugar@lists.laptop.org/msg06957.html
. There are several other situations that once fixed would give us
wins in the order of hundreds of milliseconds with the corresponding
wins in memory usage, such as importing modules that are never used.

The move from PyGtk to introspection will mean we stop loading several
dozens of classes that we never end up using, but there's lots to win
in the Python std library as well.

Regards,

Tomeu

> and my instinct tells me that just
> loading a Python source file will reference almost all the shared pages
> because they are mostly used to store already loaded modules. What I finally
> did not do is to actually check the hypothesis that most of the heap is
> filled by strings which are the identifiers in the loaded Python modules. My
> goal was to somehow make identifiers constants and just readonly-mmap them
> into the process (replace the pathetic .pyc loading mechanism). Note that my
> plan was just a little subset of the .jar -> .dex converter.
>
> Second, Python has a special memory manager which works with buckets so
> there are separate heaps for different object sizes. Somehow this special
> memory manager is turned off in release mode and it uses the standard C heap
> for some reason. Either it is a bug or just it turned out to be faster (more
> work was put into glibc) I do not know, but handling the linked free list
> can dirty shared pages as well or I am just mistaken...
>
> Third, the fact that Python is a dynamic language does not help any
> prefetching or memory sharing. I am not too convicted either that this
> dynamic nature is used at all in the Sugar codebase but you know I cannot
> program in Python and frankly I do not feel the need to learn another
> language. Just now, at my age of 34, I finally gave up and learned LISP
> (more specifically clojure) and I hope that it will be the last programming
> language I will have to learn (other than assembly languages of course)...
> :) Now this point is interesting because if you thought that the Dalvik VM
> could run the Sugar codebase via Jython then it just will not work. The
> Dalvik VM just cannot load .jar files and Jython just generates them on the
> fly because of the dynamic nature of Python.
>
> Fourth, moving Python (theoretically) to a GC environment (instead of
> reference counting) also would not work if the GC is compacting because it
> would also dirty the shared pages. So a mark and sweep nonmoving GC with
> separately stored "visited" bits is the only feasible solution. Now guess
> what the Dalvik VM does?
> For more info (45 min + questions):
> http://www.youtube.com/watch?v=ptjedOZEXPM
>
> So *my* conclusion is that solving this sharing problem is *very* hard. I am
> not sure that simply translating all activities from Python to Java would
> take more time.
>
> Another interesting thing is that meantime the unladen-swallow project
> progresses (just more slowly than planned). Their plan is to make it the
> default Python runtime so if it will happen (I cannot comment on that) then
> the Python VM will use even more memory (though it will be faster) so Sugar
> will be even less interesting on the myriad of low spec cheap ARM tablets
> which will flood the market soon.
>
> I think that is all I can say about this subject so I just finish it here.
>
>
>
>