Reduce memory footprint with large collections

[centic]

I would like to discuss ways how we can reduce the amount of memory that is used by Jajuk when collections become large. E.g. my collection contains around 14500 titles and with this Jajuk consumes 115MB of memory. And I am not a big collector in any way, just took some of my CDs and ripped them over the years...

For users with small systems, this becomes a problem quickly. Especially when collections become very large, Jajuk become unusable because of memory consumption.

How do we measure which parts use up all the memory? Any good tool for that?

[bflorat]

Indeed, Jajuk uses a lot of memory for three main reasons:

1- Jajuk doesn't use lazy loading or a database but load all the database at startup from an XML file. This takes about 20 to 30% of the total memory . Note that IMO, this design is still the best so far for our need but I may be wrong.

2- Jajuk uses advanced GUI, for instance the Substance look and feel that takes 10Mb alone and many advanced widgets from swingx. And we display a *lot" of information in many tables... Don't forget that we bring to users many advanced features that works on the entire collection (like instant search box)

3- Jajuk heavily works on image manipulation (catalog views, thumbs, covers...) : this takes the largest part of the memory

We already worked deeply on performances, especially on 1.4 release and we did a lot of advanced memory optimization (for exemple, thumbs are created on a separated JVM). I do think that we could still gain few % but not at important levels. Note as well that memory and CPU optimization are contradictory most of the time (for ie, caching increase the memory but decrease CPU consumption)

Moreover, I feel that given the feature we bring (instant search on entire collection, thousands of images...), the memory used is pretty low comparated to many others apps. Standard PC comes now with 2Go of RAM and soon 4Go...

You're welcomed to hunt memory optimizations using the *excellent* tools from netbeans: VisualVM : terrific !

[bflorat]

I forget: you told about problems on huge collection. I never heard about them. Please create detailed tickets if reproductable.

[anonymous]

I did start some review and posted some patches to the mailing list for review. Furthermore I updated  http://www.jajuk.info/index.php/Performance_tips#Memory_Usage with some more suggestions.

[bflorat]

Thanks for the feedback. I just removed from the wiki few things. We can't seriously ask a user to delete playlist files to use a jukebox ! (BTW, future features will allow user to create playlist files massively). I gonna check your patches but to my knowlege, the playlist data in memory should not take more than few KB.

[centic]

Over time I did some more analysis on this, here some additional findings:

- Views are created separatedely for each perspective. So if a view holds a list of items and is used in multiple perspectives, it will duplicate it's content while it should still display equally in all perspectives anyway...

- Things like dialogs or other GUI elements are not fully garbage collected sometimes because some internal Swing or Substance components still holds onto it. This means that e.g. if you use some of the dialogs a lot you will accumulate more and more memory. Not sure if there is a way to free up this when a dialog is closed.

- Things that increase Memory usage a lot are: Large Collections (now at 25000 titles for me), large Queues (I usually have a queue that has thousands of entries), Views that hold a lot of images like Album thumbs, ...

[bflorat]

- Views are created separatedely for each perspective. So if a view holds a list of items and is used in multiple perspectives, it will duplicate it's content while it should still display equally in all perspectives anyway...

Do you mean the models ? if so, few of them are actually duplicated : the files table view model is different from the tracks table view one, same between tree view ones. Of course, if you instanciate several time the same view, you'll get the same model several times but it is not a problem as:

• we want to keep views independant and manage there own models
• anyway, it's just a theorical case as there's no need for displaying several time the same table or tree view inside the same perspective

- Things like dialogs or other GUI elements are not fully garbage collected sometimes because some internal Swing or Substance components still holds onto it. This means that e.g. if you use some of the dialogs a lot you will accumulate more and more memory. Not sure if there is a way to free up this when a dialog is closed.

How do you know that ? using a profiler ? maybe should we try to reproduce through simple standalone cases and report an issue to substance if we reproduce.

- Things that increase Memory usage a lot are: Large Collections (now at 25000 titles for me), large Queues (I usually have a queue that has thousands of entries), Views that hold a lot of images like Album thumbs, ...

This is actually the main problem (the collection itself only keeps in memory around 3 or 4x the XML collection file size). However, the images takes large amount of memory. We already optimized mush of this by flushing resources but we may have forgotten some (don't think so). It's just that we deal with large number of images so we use a lot of memory, others image manipulation apps have the same "problem" if we can call this a problem. Note : when tring to save memory on images, we need to be very careful to avoid issues because it can comes with image refreshing problems (for instance, when changing a cover in catalog view, the same old image is kept). Images manipulation in java is a complex area (partially because they're different API generations around the time) so we may want to check if our image manipulation patterns are correct or find a java imaging / java 2D expert for an audit.

[bflorat]

Many actions have been token since then, closing the discussion