Topic: Memory Leak in AMD64 and RPi builds (Read 22620 times)

bob · « **on:** October 15, 2015, 01:36:56 pm »

Quote from: UdoS on October 15, 2015, 09:23:18 am

I turned off the spectrum analyzer and it seems to low down using memory. I run the system now for more than 24h and it build up to 771Mb, far less than before, but still counting up.
The heap increases when you start a new song, sometime more, sometimes less.

I can reproduce this on ubuntu 15.10 even with the analyzer off.
Hoping it's a generic amd64 problem (since it doesn't happen on my i386 wheezy system).
Will do some testing on the build system and go from there...

UdoS · « **Reply #1 on:** October 16, 2015, 09:19:53 am »

Quote from: bob on October 15, 2015, 01:36:56 pm

I can reproduce this on ubuntu 15.10 even with the analyzer off.
Hoping it's a generic amd64 problem (since it doesn't happen on my i386 wheezy system).
Will do some testing on the build system and go from there...

I run it for 24h now with spects turned back on and it almost 18x memory usage (12GB)

bob · « **Reply #2 on:** October 16, 2015, 10:31:17 am »

Quote from: UdoS on October 16, 2015, 09:19:53 am

I run it for 24h now with spects turned back on and it almost 18x memory usage (12GB)

I have not been able to duplicate the issue on Debian Jessie AMD64 but did on Ubuntu 15.10 AMD64.

This leads me to suspect a library incompatibility. Would you tell us again what system you are running on?

UdoS · « **Reply #3 on:** October 16, 2015, 11:03:29 am »

Quote from: bob on October 16, 2015, 10:31:17 am

I have not been able to duplicate the issue on Debian Jessie AMD64 but did on Ubuntu 15.10 AMD64.

This leads me to suspect a library incompatibility. Would you tell us again what system you are running on?

Suse 13.2-64

A list of used libs is attached below

bob · « **Reply #4 on:** October 16, 2015, 05:05:04 pm »

Please check and see if this resolves memory usage on non-debian distros.

UdoS · « **Reply #5 on:** October 17, 2015, 03:04:15 pm »

Quote from: bob on October 16, 2015, 05:05:04 pm

Please check and see if this resolves memory usage on non-debian distros.

I run MC now for 1h and memory usage jumped form 88Mb to 884Mb (spect. enabled)

Mike Noe · « **Reply #6 on:** October 18, 2015, 11:12:57 am »

fwiw, I'm seeing the same issue here on TumbleWeed x86-64 Plasma5. Been running the x86-64 build for a little while but the i586 version was eating memory too, since a few builds back. I've seen about 500MB per 12 hrs of up time and I have a couple of MCWS clients hitting it every 10 seconds or so.

RemyJ · « **Reply #7 on:** October 18, 2015, 01:23:23 pm »

Giving 21 a try under Linux again...

If you move or resize the window, the app hangs on exit.

PrinterPrinter · « **Reply #8 on:** October 19, 2015, 07:40:57 am »

Hello,
I have MC21 running on RPI and generally like it very much.
I noticed that after a while (12 hours?) or running MC21 its memory footprint keeps growing until the Pi runs out of memory and then crashes. I had a similar issue with MC20 - nothing else is running on the Pi.

Thank you

mwillems · « **Reply #9 on:** October 19, 2015, 07:48:41 am »

Have you built all your thumbnails? I've had similar crashes while my library was building thumbnails, but once all thumbs are built I have very stable memory usage. If you could try pre-building all your thumbs as a diagnostic that could help (the option is under library & folders-->build missing thumbnails)

Even if that's not the issue, you'll see improved responsiveness of the interface (and gizmo/jremote) with all the thumbs pre-built so it won't be a waste of time either way. When you change major versions (i.e. from 20 to 21) the thumbs don't get brought across so you're starting fresh with 21.

PrinterPrinter · « **Reply #10 on:** October 19, 2015, 08:06:37 am »

Quote from: mwillems on October 19, 2015, 07:48:41 am

Have you built all your thumbnails? I've had similar crashes while my library was building thumbnails, but once all thumbs are built I have very stable memory usage. If you could try pre-building all your thumbs as a diagnostic that could help (the option is under library & folders-->build missing thumbnails)

Even if that's not the issue, you'll see improved responsiveness of the interface (and gizmo/jremote) with all the thumbs pre-built so it won't be a waste of time either way. When you change major versions (i.e. from 20 to 21) the thumbs don't get brought across so you're starting fresh with 21.

Thanks I'll do it.
I don't have any local media on the Pi - it's all streaming from a bigger windows machine running MC20 - so I still need to build the thumbs on the Pi? Or on teh windows machine? Or both?

Thank you!

Update - I don't have option like that - I can see one on my local Pi (under configure auto import) but it complains I have no media folders checked - because I'm playing a library from another server...

mwillems · « **Reply #11 on:** October 19, 2015, 08:20:55 am »

Do it on the Pi instance. When a JRiver instance displays a view it has to create thumbnails to display from teh album cover art. Creating a thumbnail to display requires CPU, memory, and disk access, and in the case of a remote client it requires network bandwidth too. For this reason, JRiver caches the thumbnails on disk locally once they're built so the display stays responsive (which is one of the ways that JRiver is able to handle such large libraries while every other player chokes).

On a full spec PC with a fast network connection the resource load is pretty trivial (thumbs are created as needed pretty quickly), but on the pi, you'll notice that if you open a view you haven't looked at before you'll see a lot of those blue musical notes and then all the cover art will gradually fill in over several seconds (sometimes several tens of seconds). The time it takes to fill in is the Pi struggling to pull the cover art into memory across the network downscale it to thumbnail size and then display it. If you build them all in advance on the pi, they'll just be "there" already so the only bottleneck is disk access. Gizmo and JRemote pull from the same thumbnail cache, so pre-creating them will help there too.

In my experience pre-building thumbs drastically improves view responsiveness on the Pi so it's a good investment of time no matter what. Set thumbnailing priority to "low" (to minimize memory usage) and leave it to run overnight; it may crash (due to the memory leak) but progress won't be lost, just run it again if that happens and it will pick up where it let off. You can check under services/reporter to see how many thumbs are currently built.

And report back about whether that resolves the memory leak once it's done.

bob · « **Reply #12 on:** October 19, 2015, 10:45:15 am »

I think you non-debian users are running into library incompatibilities (since I can't reproduce the memory issue on either i386 or AMD64 debian) and I'm not sure how much farther we can go down the path to try and make this work for you.
The AMD64 build has about every library it can have to make it work on other distros linked into MC's private library path.

mwillems · « **Reply #13 on:** October 19, 2015, 11:05:44 am »

Quote from: bob on October 19, 2015, 10:45:15 am

I think you non-debian users are running into library incompatibilities (since I can't reproduce the memory issue on either i386 or AMD64 debian) and I'm not sure how much farther we can go down the path to try and make this work for you.
The AMD64 build has about every library it can have to make it work on other distros linked into MC's private library path.

FWIW I run the AMD64 build on Arch and see no memory leak.

Bob, just a wild thought, but someone reported a memory leak over on ARM as well this morning on raspbian. Do you think the two might be connected? I speculated in that thread that it might be related to the thumbnailing memory leak on ARM (which is improved but not fixed). Do you think the leak in this thread could be connected to thumbnailing? My amd64 instance where I see no memory leak already has all of its thumbnails built so that's one differential. That would also explain why you might see it on newer installs, but not longstanding ones.

Anybody else running the 64 bit version on Arch? Anyone seeing the memory leak?

mwillems · « **Reply #14 on:** October 19, 2015, 11:12:20 am »

Quote from: PrinterPrinter on October 19, 2015, 08:06:37 am

Update - I don't have option like that - I can see one on my local Pi (under configure auto import) but it complains I have no media folders checked - because I'm playing a library from another server...

Sorry I mentioned the wrong menu location. It's under Options-->Tree and View-->Thumbnails-->Build missing thumbnails

PrinterPrinter · « **Reply #15 on:** October 19, 2015, 12:30:53 pm »

Quote from: mwillems on October 19, 2015, 11:12:20 am

Sorry I mentioned the wrong menu location. It's under Options-->Tree and View-->Thumbnails-->Build missing thumbnails

Ok,
Thanks.

Under services reporter it says all Thumbnails are built (100%). I did set it up as you suggested anyway - thank you.

BTW
Not sure if its related but I found something funny - when the spectrum analyser runs in the small window in the tool bar (that says the name of the file, time etc) it causes the Pi to use about 25% of its CPU when playing music - if I turn it off the CPU usage drops to around 5% when playing music...

Not sure if its related. I play mostly FLAcs and have a very modest EQ

bob · « **Reply #16 on:** October 19, 2015, 12:52:41 pm »

Quote from: mwillems on October 19, 2015, 11:05:44 am

FWIW I run the AMD64 build on Arch and see no memory leak.

Bob, just a wild thought, but someone reported a memory leak over on ARM as well this morning on raspbian. Do you think the two might be connected? I speculated in that thread that it might be related to the thumbnailing memory leak on ARM (which is improved but not fixed). Do you think the leak in this thread could be connected to thumbnailing? My amd64 instance where I see no memory leak already has all of its thumbnails built so that's one differential. That would also explain why you might see it on newer installs, but not longstanding ones.

Anybody else running the 64 bit version on Arch? Anyone seeing the memory leak?

I was wondering about that when I saw the message about the memory leak on arm.
I would think that with a small library though I wouldn't see a leak, at least not a massive one.

bob · « **Reply #17 on:** October 19, 2015, 12:54:08 pm »

Quote from: PrinterPrinter on October 19, 2015, 12:30:53 pm

Ok,
Thanks.

Under services reporter it says all Thumbnails are built (100%). I did set it up as you suggested anyway - thank you.

BTW
Not sure if its related but I found something funny - when the spectrum analyser runs in the small window in the tool bar (that says the name of the file, time etc) it causes the Pi to use about 25% of its CPU when playing music - if I turn it off the CPU usage drops to around 5% when playing music...

Not sure if its related. I play mostly FLAcs and have a very modest EQ

You could turn the analyzer off to see if that's using memory. It does actually consume a fair amount of CPU on a Pi to display that.

bob · « **Reply #18 on:** October 19, 2015, 04:19:46 pm »

It seems there is a memory leak associated with playback not much with Thumbnailing as far as I can see.
I wiped my thumbnails and generated new ones for 6k tracks and used up a total of 8 megs so if there is a leak there, it's not huge.
I did leave an mp3 playlist on a test machine over the weekend and it was up to 1.8g of memory used by the time I came in today.
Testing MC20 to see if it's been there forever or if it's new.

bob · « **Reply #19 on:** October 19, 2015, 04:21:20 pm »

Quote from: mwillems on October 19, 2015, 11:05:44 am

FWIW I run the AMD64 build on Arch and see no memory leak.

Bob, just a wild thought, but someone reported a memory leak over on ARM as well this morning on raspbian. Do you think the two might be connected? I speculated in that thread that it might be related to the thumbnailing memory leak on ARM (which is improved but not fixed). Do you think the leak in this thread could be connected to thumbnailing? My amd64 instance where I see no memory leak already has all of its thumbnails built so that's one differential. That would also explain why you might see it on newer installs, but not longstanding ones.

Anybody else running the 64 bit version on Arch? Anyone seeing the memory leak?

Could you just try looping a mp3 file for local playback? I got a big leak leaving a machine running over the weekend. It might be something about my audio settings too or a leaky audio driver perhaps...

mwillems · « **Reply #20 on:** October 19, 2015, 05:34:19 pm »

Quote from: bob on October 19, 2015, 04:21:20 pm

Could you just try looping a mp3 file for local playback? I got a big leak leaving a machine running over the weekend. It might be something about my audio settings too or a leaky audio driver perhaps...

Ok good news and bad news:

The bad news: I couldn't reproduce it before because I wasn't running the latest build and didn't realize it

Now that I'm on the latest build I can reproduce the leak. It's easy to see if you're watching a memory monitor. With flac files JRiver's memory usage grows quite substantially on each track change, every single time. You can accelerate reproduction by playing a file waiting a five or six seconds and then skipping forward a track and repeat. I saw memory growth on each skip. A single album got me up to nearly a Gig of memory where MC normally never uses much more than 200M. I should mention I'm also pinging MCWS every five seconds with a script if that contributes (someone else mentioned that above). That was never previously an issue though.

The good news: My being a dope about updating inadvertently resulted in a successful bisect

I can confirm that version 21.0.07 doesn't exhibit the memory leak at all for me on Arch, no increase on track change whatsoever. So it's likely been introduced since then.

mwillems · « **Reply #21 on:** October 19, 2015, 05:36:46 pm »

I think it's new, I have JRiver running on Pi's with 60+ days of uptime that do frequent playback. I've looked at their memory logs and there's no evidence of a leak on that scale (I can post munin-graphs if that would help). Also see my comment over in the amd64 thread, based on my testing I think this came in after 21.0.7.

PrinterPrinter · « **Reply #22 on:** October 19, 2015, 08:00:08 pm »

I had experienced this memory leak back with MC20 on RPI - maybe there's some conflict in my set up?

My Pi isn't able to play 24 hours without a reboot or a crash ;-) Same behaviour in MC20

Hope this helps,

bob · « **Reply #23 on:** October 20, 2015, 11:18:09 am »

Perhaps it's something specific about the options settings.

What are each of your settings for audio playback? (track change settings, stop settings).

bob · « **Reply #24 on:** October 20, 2015, 11:19:59 am »

Quote from: mwillems on October 19, 2015, 05:34:19 pm

Ok good news and bad news:

The bad news: I couldn't reproduce it before because I wasn't running the latest build and didn't realize it Now that I'm on the latest build I can reproduce the leak. It's easy to see if you're watching a memory monitor. With flac files JRiver's memory usage grows quite substantially on each track change, every single time. You can accelerate reproduction by playing a file waiting a five or six seconds and then skipping forward a track and repeat. I saw memory growth on each skip. A single album got me up to nearly a Gig of memory where MC normally never uses much more than 200M. I should mention I'm also pinging MCWS every five seconds with a script if that contributes (someone else mentioned that above). That was never previously an issue though.

The good news: My being a dope about updating inadvertently resulted in a successful bisect I can confirm that version 21.0.07 doesn't exhibit the memory leak at all for me on Arch, no increase on track change whatsoever. So it's likely been introduced since then.

Thanks very much. What are your track change settings?

mwillems · « **Reply #25 on:** October 20, 2015, 11:34:26 am »

Quote from: bob on October 20, 2015, 11:19:59 am

Thanks very much. What are your track change settings?

On the Arch PC the track change settings are gapless for sequential album tracks (checked) and otherwise cross-fade aggressive. I should note that I see memory growth on almost every track change whether it happens normally at the end of the track (gapless) or through the skip button (crossfade). I mentioned using skip above as that seemed to speed up the demonstration of the leak, but I did notice that skipping tracks as quickly as possible without waiting for the fade in to finish didn't always result in significant memory growth. The track needs to buffer a bit to trigger it I think.

mwillems · « **Reply #26 on:** October 20, 2015, 11:40:56 am »

My settings on all my pi's are gapless for sequential album tracks (checked) and cross-fade aggressive otherwise. I'm currently running along playlist to see if I can reproduce the memory leak on 21.0.14. I definitely can't on 21.0.07

mwillems · « **Reply #27 on:** October 20, 2015, 12:23:40 pm »

Ok, I can definitely reproduce it in 21.0.14. A single FLAC album (albeit a long one) had my MC memory use jumping a few hundred MB. Stopping one album and starting another seemed to make the biggest jump. I did four albums worth of skipping around, stopping and starting a new album before RAM use got up around 700MB+ and then MC froze and crashed to desktop.

PrinterPrinter · « **Reply #28 on:** October 20, 2015, 02:38:28 pm »

I don't use gapless but do have 'do not play Silence' ticked. Cross fade aggressive - I hope it helps?

bob · « **Reply #29 on:** October 20, 2015, 02:48:09 pm »

Quote from: mwillems on October 20, 2015, 11:34:26 am

On the Arch PC the track change settings are gapless for sequential album tracks (checked) and otherwise cross-fade aggressive. I should note that I see memory growth on almost every track change whether it happens normally at the end of the track (gapless) or through the skip button (crossfade). I mentioned using skip above as that seemed to speed up the demonstration of the leak, but I did notice that skipping tracks as quickly as possible without waiting for the fade in to finish didn't always result in significant memory growth. The track needs to buffer a bit to trigger it I think.

I set my Wheezy i386 build machine to cross fade aggressive .5 sec and all of the seek, pause, stop to normal (fade-out) and I can't reproduce it.
On the AMD64 Jessie build machine though the same settings are obviously causing a leak.

mwillems · « **Reply #30 on:** October 20, 2015, 03:15:06 pm »

So you've repro'd it on the Amd64 build with Jessie? That's a step forward (I think?) Let me know what I can test to help, I have a bunch of machines in different environments and can fire up a VM if need be.

bob · « **Reply #31 on:** October 20, 2015, 03:23:35 pm »

Quote from: mwillems on October 20, 2015, 03:15:06 pm

So you've repro'd it on the Amd64 build with Jessie? That's a step forward (I think?) Let me know what I can test to help, I have a bunch of machines in different environments and can fire up a VM if need be.

Unfortunately I can also reproduce it on Jessie with 21.0.7. Are you certain that you don't see that when you install the 21.0.7 build?

Also, mine really seems to jump when switching to a new album (coverart?)

bob · « **Reply #32 on:** October 20, 2015, 05:30:10 pm »

Can you give this build a try and see if it makes a difference?
http://files.jriver.com/mediacenter/channels/v21/latest/MediaCenter-21.0.17-amd64.deb

bob · « **Reply #33 on:** October 20, 2015, 05:31:27 pm »

Can you give this build a try and see if it makes any difference?
http://files.jriver.com/mediacenter/channels/v21/latest/MediaCenter-21.0.17-armhf.deb

Mike Noe · « **Reply #34 on:** October 20, 2015, 06:17:29 pm »

Interesting...
So, on my TW setup, first run on that build starts at 96MB, which is much higher than prev builds....but, if I start playback, jump back and forth amongst albums/tracks, start, stop playback, mem usage never goes above 123MB. Then, if I stop playback and wait about 10 seconds, it drops back to ~100MB....controlling is with a MCWS client.

Then, stopped it, started it again and started playback from the MC GUI and it sits right at ~70MB.... then I "connected" my MCWS client, switched tracks and mem use jumps to 115MB...where it sits.

I'll leave it playing see what happens....

PrinterPrinter · « **Reply #35 on:** October 20, 2015, 06:51:36 pm »

Quote from: bob on October 20, 2015, 05:31:27 pm

Can you give this build a try and see if it makes any difference?
http://files.jriver.com/mediacenter/channels/v21/latest/MediaCenter-21.0.17-armhf.deb

I'm ashamed to say I managed to download the file to my Pi but have no idea how to install it - opening it in file explorer doesn't do the trick... ?

Awesome Donkey · « **Reply #36 on:** October 20, 2015, 07:52:05 pm »

In a Terminal try this if you downloaded it into the default download folder;

Code: [Select]

sudo dpkg -i ./Downloads/MediaCenter-21.0.17-armhf.deb

PrinterPrinter · « **Reply #37 on:** October 21, 2015, 04:31:04 am »

Quote from: Awesome Donkey on October 20, 2015, 07:52:05 pm

In a Terminal try this if you downloaded it into the default download folder;

Code: [Select]
sudo dpkg -i ./Downloads/MediaCenter-21.0.17-armhf.deb

Thanks!
Got it installed - looks like teh leak is still there but I'll give it a few more hours to be certain...

mwillems · « **Reply #38 on:** October 21, 2015, 07:30:03 am »

It still appears to be there, but is slower than it was in .14

mwillems · « **Reply #39 on:** October 21, 2015, 07:32:34 am »

Quote from: bob on October 20, 2015, 03:23:35 pm

Unfortunately I can also reproduce it on Jessie with 21.0.7. Are you certain that you don't see that when you install the 21.0.7 build?

Quite certain, I downgraded to test. On two different Arch 64-bit machines the 21.0.7 build doesn't show any leak on track or album changes.

Quote

Also, mine really seems to jump when switching to a new album (coverart?)

It does jump on album changes, but if you change albums very quickly in succession it doesn't grow as much for me as when you change albums, wait a bit, change again, etc. Maybe I'm just seeing things, but it seems like it has to do with buffering.

21.0.17 still has the leak, but it's slower for me now.

PrinterPrinter · « **Reply #40 on:** October 21, 2015, 10:11:35 am »

Yep the leak is there...

bob · « **Reply #41 on:** October 22, 2015, 09:57:25 am »

Quote from: mwillems on October 21, 2015, 07:32:34 am

Quite certain, I downgraded to test. On two different Arch 64-bit machines the 21.0.7 build doesn't show any leak on track or album changes.

It does jump on album changes, but if you change albums very quickly in succession it doesn't grow as much for me as when you change albums, wait a bit, change again, etc. Maybe I'm just seeing things, but it seems like it has to do with buffering.

21.0.17 still has the leak, but it's slower for me now.

Can we try some things to narrow this down? I left my i386 wheezy running playing flacs for 2 days straight without any significant memory growth (sitting where it was 2 days ago, 144M). I had to restart the Jessie AMD64 test this morning so I don't have data on that.
I'm in standard view mode with your track transition settings, no DSP studio involved, the internal front: intel audio device.
With the file list showing in the main frame and only the small coverart in the action window.

bob · « **Reply #42 on:** October 22, 2015, 10:25:01 am »

I merged the memory leak topics together into this new thread.

bob · « **Reply #43 on:** October 22, 2015, 05:33:47 pm »

Can you test this one please ARM people.

http://files.jriver.com/mediacenter/channels/v21/latest/MediaCenter-21.0.17-3-armhf.deb

Hendrik · « **Reply #44 on:** October 22, 2015, 05:51:59 pm »

Bob, if you can ever reproduce it, you should throw valgrind at it. Benefit from all these linux tools.

mwillems · « **Reply #45 on:** October 22, 2015, 06:37:26 pm »

Quote from: bob on October 22, 2015, 05:33:47 pm

Can you test this one please ARM people.

http://files.jriver.com/mediacenter/channels/v21/latest/MediaCenter-21.0.17-3-armhf.deb

Preliminary testing on an RPI suggests that this might have plugged the leak. I'm going to let it play overnight to be sure.

mwillems · « **Reply #46 on:** October 23, 2015, 07:35:49 am »

Quote from: bob on October 22, 2015, 05:33:47 pm

Can you test this one please ARM people.

http://files.jriver.com/mediacenter/channels/v21/latest/MediaCenter-21.0.17-3-armhf.deb

No dice. I left it playing overnight and it went from around 200M to 550M. It's much slower than it was, but it's still leaking steadily.

bob · « **Reply #47 on:** October 23, 2015, 10:18:31 am »

Quote from: Hendrik on October 22, 2015, 05:51:59 pm

Bob, if you can ever reproduce it, you should throw valgrind at it. Benefit from all these linux tools.

I did throw valgrind at it. The output was pretty crazy. I'm not sure it can follow things that we do (in JRString for instance).

bob · « **Reply #48 on:** October 23, 2015, 05:09:09 pm »

I think I've got a handle on the leak in ARM, hopefully have a new build on Monday.
The AMD64 build is another story. It's really crazy for me. 21.0.7 definitely leaks bad on the machine that produces it. Still working on that.

bob · « **Reply #49 on:** October 29, 2015, 12:12:39 pm »

Would someone try one of the latest arm builds playing a local list of preferable small files (so it has to transition through tracks frequently) and once playing, disconnect the network cable from their RPi and see if the leak still happens?

INTERACT FORUM

Author Topic: Memory Leak in AMD64 and RPi builds (Read 22620 times)