Topic: JRiver on Raspberry Pi has a memory leak? (Read 9054 times)

DrKNo · « **on:** January 22, 2020, 03:46:17 am »

I have had this issue for a long while now, but wanted to do further research before I posted. I tested on a RasPI 3b+ with Stretch and Buster, each with MC 24, 25 and 26. My media network consists of a central library server (Intel NUC, media lies on a NAS) and two RasPIs in Kitchen and Bedroom connected to a USB-C speaker, and using WiFi for data. The RasPIs run JRiver MediaCenter via tremote (i.e. they load the library of the server) and they show up as zones there.

After restart everything runs fine, and stays that way as long as I don't play anything. As soon as I start playing, RAM usage of MC slowly increases on the pi, until it can't keep up, starts stuttering, and finally dies. MC starts out with some 240MB of RAM and uses > 800MB towards the end.

I can usually play between 30 minutes and 1 hour of music before that happens. Currently, this prohibits me from using remote zone... any idea what causes this and how to fix it?

JimH · « **Reply #1 on:** January 22, 2020, 06:25:47 am »

Turn off audio analysis and import.

DrKNo · « **Reply #2 on:** January 22, 2020, 11:15:44 am »

Sorry for being dense: Turn off analysis and automatic file import on the RasPI? isn't that managed by the server? I'll try it once I'm home and report back. Thanks for the quick reply, Jim.

DrKNo · « **Reply #3 on:** January 22, 2020, 12:39:28 pm »

No success, unfortunately. I made a screenshot of the settings and attached them. RAM still runs full.
RAM usage clearly increases at the start of a new song, then continues for a while (while it is being streamed?) and then stops for a while before it continues to rise with the next song.
There are no songs in the local library, only in the remote one on the server.

JimH · « **Reply #4 on:** January 22, 2020, 01:20:17 pm »

What are you playing?

mwillems · « **Reply #5 on:** January 22, 2020, 01:37:10 pm »

Under media network-->client options there are three options. Are any or all of those checked? If so, try unchecking the sync library automatically (or something like that) check box.

I've had a longstanding memory leak in client server situations with that option checked; every time the client syncs the library a little more memory is used until the client crashes. It presents exactly like what you're seeing. I can reproduce it instantly just by clicking the refresh view button in standard view, but bob hasn't been able to reproduce it, so I assumed it was some idiosyncrasy of my network. I see it with all linux clients, not just raspberries, but it's much more obvious on raspberries because of the limited RAM.

Try disabling that option and see if that's your issue.

DrKNo · « **Reply #6 on:** January 22, 2020, 03:26:18 pm »

Quote from: mwillems on January 22, 2020, 01:37:10 pm

If so, try unchecking the sync library automatically (or something like that) check box.

That did it! RAM usage increases with a new song, but is freed after the song ends, MC is now stably using ~270MB of RAM for half an hour, it usually would have been in the sixhundereds now. Thank you so much! I suppose library sync is unnecessary for me anyway since I am using the raspberry PIs as playback endpoints only.

Quote from: JimH on January 22, 2020, 01:20:17 pm

What are you playing?

FLAC and MP3. FLACs might have seemed to increase the RAM usage more, but I would have to verify.

I'm happy to investigate/contribute if possible. Jim, I could send you an SD image if that helped? My server is reachable from the Internet, so we could even test that.

JimH · « **Reply #7 on:** January 22, 2020, 03:29:15 pm »

We'll take a look. Thanks, mwillems.

bob · « **Reply #8 on:** January 23, 2020, 02:48:33 pm »

Are you running 26.0.19?
I'm going to upload a new build with a fix for that so when you see it if you could test it with library sync back on that would be good.
Thanks.

DrKNo · « **Reply #9 on:** January 24, 2020, 02:50:13 am »

Happy to. Update it both on server and on RasPi? I'm not sure I get to test it before Wednesday, but I'll try my best. What build number should I check? Or did you mean I should test 26.0.19?

Edit: Nevermind, saw the fix in 26.0.20, I'll report as soon as I get the chance.

mwillems · « **Reply #10 on:** January 24, 2020, 09:08:27 am »

Bob, I'm testing v. 26.0.20 and still seeing significant memory growth with the auto sync setting enabled. It increases slowly on every track change (it seems to grow about 1GB every five tracks or so for me), but I can force memory growth instantly by changing between library views rapidly (i.e. clicking back and forth between artist and genre, etc.).

My server is running Debian Stable (Buster), and I tested with clients running Debian Stable and Arch Linux.

bob · « **Reply #11 on:** January 24, 2020, 02:50:20 pm »

Quote from: mwillems on January 24, 2020, 09:08:27 am

Bob, I'm testing v. 26.0.20 and still seeing significant memory growth with the auto sync setting enabled. It increases slowly on every track change (it seems to grow about 1GB every five tracks or so for me), but I can force memory growth instantly by changing between library views rapidly (i.e. clicking back and forth between artist and genre, etc.).

My server is running Debian Stable (Buster), and I tested with clients running Debian Stable and Arch Linux.

Thanks, Hendrik also thinks it's still an issue... We'll look some more.

Hendrik · « **Reply #12 on:** January 31, 2020, 05:28:07 am »

I've been trying to reproduce this and check for memory leaks, but I could not find anything substantial, or even any evidence of misbehavior.
I did find a few tiny memory leaks, which I fixed, but all in the range of kilobytes, not mega or even gigabytes.

Syncing with the server does produce an increase in memory requirement - but only a one-time use.
When you sync the library back to the server, MC has to determine a "delta" of what actually changed. We don't store a change date for every single field (that would be insane), so instead we compare our local library to the remote library. To do this, it has to load the entire library completely into memory. This is usually not done outside of syncing, or until you have viewed every field of every file in your library manually. On a very large database, this can be quite some memory.

Once this data is loaded once, it should not grow further beyond that. A second sync process won't load it again, since its already loaded.

All that said, I also found one case where memory was being wasted that was only used conditionally, so I've cleaned that up and a fully loaded library, like in the sync process, should now use significantly less memory.

If this is still going on with the next version, and one of you can reproduce it reliably on a x86_64 system, we might ask you to collect some information for us.

mwillems · « **Reply #13 on:** January 31, 2020, 09:05:35 am »

Quote from: Hendrik on January 31, 2020, 05:28:07 am

I've been trying to reproduce this and check for memory leaks, but I could not find anything substantial, or even any evidence of misbehavior.
I did find a few tiny memory leaks, which I fixed, but all in the range of kilobytes, not mega or even gigabytes.

Syncing with the server does produce an increase in memory requirement - but only a one-time use.
When you sync the library back to the server, MC has to determine a "delta" of what actually changed. We don't store a change date for every single field (that would be insane), so instead we compare our local library to the remote library. To do this, it has to load the entire library completely into memory. This is usually not done outside of syncing, or until you have viewed every field of every file in your library manually. On a very large database, this can be quite some memory.

Once this data is loaded once, it should not grow further beyond that. A second sync process won't load it again, since its already loaded.

All that said, I also found one case where memory was being wasted that was only used conditionally, so I've cleaned that up and a fully loaded library, like in the sync process, should now use significantly less memory.

If this is still going on with the next version, and one of you can reproduce it reliably on a x86_64 system, we might ask you to collect some information for us.

Thanks Hendrik! I've been experiencing this reliably on all linux platforms (amd64 and arm) for a few major versions, so I'll be happy to collect info if I can still reproduce it after the next build.

bob · « **Reply #14 on:** January 31, 2020, 02:43:54 pm »

Quote from: mwillems on January 31, 2020, 09:05:35 am

Thanks Hendrik! I've been experiencing this reliably on all linux platforms (amd64 and arm) for a few major versions, so I'll be happy to collect info if I can still reproduce it after the next build.

[Edited]
I can verify this happens in Linux and ~~Mac~~ at any rate back into MC25, Hendrik fixed some things but it's still happening for me. It seems to bump up on every track. I tried MP3 and M4A lossless.
On the Mac playing back files from my server it's gone from 100MB memory usage to 150MB in an hour or so and it may have stopped there...
On a Pi running Buster from 150MB to 180MB in about the same time, on an IdPi MC25 similar behavior.
On my AMD64 box after several hours it's up to 429MB but after a few hours that went back down to 320MB.

Not totally sure what's going on, letting the test run over the weekend...

bob · « **Reply #15 on:** February 03, 2020, 01:52:49 pm »

Would you all give 26.0.22 a try please?

Mike Noe · « **Reply #16 on:** February 03, 2020, 06:00:01 pm »

I hadn't noticed this issue on my Pi4 here, but pre-22 it was starting at about 90MB and after a few hours would hit 200MB, then hold there.

With 22, it starts about the same, but after a couple of hours, tops out at 110MB, same playback/connection conditions. So that looks nice. ("RSS" number from the Raspbian Task Manager)

This is with two or three MCWS clients polling and whatnot, with the Pi (running Raspbian) playing back to just one USB Audio device (zone) with a fairly light DSP setup (Output Format/Volume Leveling/Adaptive Vol/Equalizer).

DrKNo · « **Reply #17 on:** February 04, 2020, 01:30:07 am »

Raspberry PI on .22 shows the same behaviour for me (Server is .18, Windows 10). No DSP on the Pi, RAM usage stops rising once AutoSync is deactivated.

bob · « **Reply #18 on:** February 04, 2020, 10:14:18 am »

Ran 26.0.22 overnight on a Pi4 with sync on and it stabilized at 180MB before I left yesterday and is still at that (and has been playing continuously since then) this morning.

If anyone is still seeing memory rise, more details are going to be necessary.

mwillems · « **Reply #19 on:** February 04, 2020, 05:08:20 pm »

22 seems a better for me in preliminary testing. I can still force big spikes in RAM usage by mashing the refresh button repeatedly, but some/most of the RAM gets released later. Ditto with rapid view changes.

I can get RAM usage to stick up at around 1G on my amd64 systems, but no higher even after extended playback. On armhf, jriver uses about 200MB before I play anything, I could force the pi up to around 500MB (via refreshing and view switching) but it would sink back down to 400MB at steady state, and I couldn't immediately get it much higher or to trigger the OOM killer.

That's a big improvement so far. I plan to leave a system or two playing overnight to see if there's still a slow leak, but so far so good here.

bob · « **Reply #20 on:** February 04, 2020, 06:07:49 pm »

Quote from: mwillems on February 04, 2020, 05:08:20 pm

22 seems a better for me in preliminary testing. I can still force big spikes in RAM usage by mashing the refresh button repeatedly, but some/most of the RAM gets released later. Ditto with rapid view changes.

I can get RAM usage to stick up at around 1G on my amd64 systems, but no higher even after extended playback. On armhf, jriver uses about 200MB before I play anything, I could force the pi up to around 500MB (via refreshing and view switching) but it would sink back down to 400MB at steady state, and I couldn't immediately get it much higher or to trigger the OOM killer.

That's a big improvement so far. I plan to leave a system or two playing overnight to see if there's still a slow leak, but so far so good here.

Thanks for the feedback, it really helps.

mwillems · « **Reply #21 on:** February 05, 2020, 08:30:52 am »

So I spoke too soon. Overnight, my amd64 test system wound up at around 4GB of ram by morning and still climbing; the rate of memory climb is definitely slower now, so that's something. My armhf test system seems fine though, holding steady between 400 and 500MB, although it briefly climbed a bit then settled back down.

Let me know what I can do to test or provide further info.

bob · « **Reply #22 on:** February 05, 2020, 03:03:39 pm »

Quote from: mwillems on February 05, 2020, 08:30:52 am

So I spoke too soon. Overnight, my amd64 test system wound up at around 4GB of ram by morning and still climbing; the rate of memory climb is definitely slower now, so that's something. My armhf test system seems fine though, holding steady between 400 and 500MB, although it briefly climbed a bit then settled back down.

Let me know what I can do to test or provide further info.

What kind of material are you playing?
Are you using any DSP studio effects?
Local or DLNA playback zone?

Thanks.

mwillems · « **Reply #23 on:** February 05, 2020, 09:13:36 pm »

Quote from: bob on February 05, 2020, 03:03:39 pm

What kind of material are you playing?
Are you using any DSP studio effects?
Local or DLNA playback zone?

Thanks.

Playing FLACs and MP3s, the only DSP effects in use are volume leveling, and the playback is local to the client (i.e. the client is starting playback to one of its local zones).

bob · « **Reply #24 on:** February 06, 2020, 12:14:14 pm »

Quote from: mwillems on February 05, 2020, 09:13:36 pm

Playing FLACs and MP3s, the only DSP effects in use are volume leveling, and the playback is local to the client (i.e. the client is starting playback to one of its local zones).

Thanks.
What view is showing during that local playback?

mwillems · « **Reply #25 on:** February 06, 2020, 01:08:00 pm »

Quote from: bob on February 06, 2020, 12:14:14 pm

Thanks.
What view is showing during that local playback?

The "recent" view, typically, which I think is a stock view? If not I can post the specification for it.

bob · « **Reply #26 on:** February 06, 2020, 01:30:03 pm »

Quote from: mwillems on February 06, 2020, 01:08:00 pm

The "recent" view, typically, which I think is a stock view? If not I can post the specification for it.

No that's fine.
What do you have set in Media Network Client Options (when connected to a Library Server) ->Audio conversion?

Hendrik · « **Reply #27 on:** February 06, 2020, 02:18:12 pm »

Also, do you use last.fm or any such things that may run during playback that may not be default?

mwillems · « **Reply #28 on:** February 06, 2020, 03:08:31 pm »

Quote from: bob on February 06, 2020, 01:30:03 pm

What do you have set in Media Network Client Options (when connected to a Library Server) ->Audio conversion?

Conversion is set to "don't convert audio"

Quote from: Hendrik on February 06, 2020, 02:18:12 pm

Also, do you use last.fm or any such things that may run during playback that may not be default?

I don't run last.fm and have no online services configured under the "Services" option menu.

bob · « **Reply #29 on:** February 06, 2020, 03:54:20 pm »

Quote from: mwillems on February 06, 2020, 03:08:31 pm

Conversion is set to "don't convert audio"

I don't run last.fm and have no online services configured under the "Services" option menu.

Ok, I've been running that configuration for the last couple of hours and memory use is stable at 330Megs.

What memory use are you looking at? I'm viewing RES in top.

JimH · « **Reply #30 on:** February 06, 2020, 03:57:12 pm »

Different desktop?

mwillems · « **Reply #31 on:** February 06, 2020, 04:05:49 pm »

Quote from: bob on February 06, 2020, 03:54:20 pm

Ok, I've been running that configuration for the last couple of hours and memory use is stable at 330Megs.

What memory use are you looking at? I'm viewing RES in top.

I'm also looking at RES in top.

Quote from: JimH on February 06, 2020, 03:57:12 pm

Different desktop?

This is on Gnome, but I can reproduce it on xfce too (which I think is bob's desktop of choice?)

mwillems · « **Reply #32 on:** February 06, 2020, 04:13:00 pm »

If it would help I could find a way to submit the server and client library backups?

bob · « **Reply #33 on:** February 06, 2020, 04:16:25 pm »

Quote from: mwillems on February 06, 2020, 04:05:49 pm

I'm also looking at RES in top.

This is on Gnome, but I can reproduce it on xfce too (which I think is bob's desktop of choice?)

I'm noticing the VIRT in top going up continuously on amd64 but it doesn't seem that really means anything.

mwillems · « **Reply #34 on:** February 06, 2020, 04:21:10 pm »

Quote from: bob on February 06, 2020, 04:16:25 pm

I'm noticing the VIRT in top going up continuously on amd64 but it doesn't seem that really means anything.

Right now after an hour or so of playback, my VIRT is 5.5 Gb and my RES is 1.5 Gb. Is there an upper limit to how much Virt MC can map?

mwillems · « **Reply #35 on:** February 06, 2020, 04:25:03 pm »

So I notice that the main gain in RAM during playback occurs between 30-40 seconds into playing the song, like clockwork. Is it trying to report back play statistics at that point?

bob · « **Reply #36 on:** February 06, 2020, 04:31:04 pm »

Quote from: mwillems on February 06, 2020, 04:21:10 pm

Right now after an hour or so of playback, my VIRT is 5.5 Gb and my RES is 1.5 Gb. Is there an upper limit to how much Virt MC can map?

I don't think so. By comparison my Firefox is using 26.9G VIRT right now (I've got a lot of open tabs).
The one that counts is the RES. 1.5 is still a lot.

Matt will check into the stats idea. In the meantime I sent you a PM...

Matt · « **Reply #37 on:** February 06, 2020, 05:28:33 pm »

Quote from: mwillems on February 06, 2020, 04:25:03 pm

So I notice that the main gain in RAM during playback occurs between 30-40 seconds into playing the song, like clockwork. Is it trying to report back play statistics at that point?

Do you have Last.fm enabled? Anything else you can think of? Thanks.

mwillems · « **Reply #38 on:** February 07, 2020, 09:07:28 am »

Quote from: Matt on February 06, 2020, 05:28:33 pm

Do you have Last.fm enabled? Anything else you can think of? Thanks.

Last.fm is not enabled. I've tried switching on and off a lot of things. I haven't been able to resolve or mitigate the issue with the auto-sync enabled, but disabling auto-sync immediately solves the problem.

Quote from: bob on February 06, 2020, 04:31:04 pm

Matt will check into the stats idea. In the meantime I sent you a PM...

I tried the build you sent, but I'm still seeing significant memory growth (possibly even faster!) with that build. After listening to one FLAC album mediacenter on the client is using 2.5Gb of memory. I installed it on both the client and the server if that's meaningful (server's only using 600MB, and that's stable more or less).

bob · « **Reply #39 on:** February 07, 2020, 09:15:46 am »

Quote from: mwillems on February 07, 2020, 09:07:28 am

Last.fm is not enabled. I've tried switching on and off a lot of things. I haven't been able to resolve or mitigate the issue with the auto-sync enabled, but disabling auto-sync immediately solves the problem.

I tried the build you sent, but I'm still seeing significant memory growth (possibly even faster!) with that build. After listening to one FLAC album mediacenter on the client is using 2.5Gb of memory. I installed it on both the client and the server if that's meaningful (server's only using 600MB, and that's stable more or less).

Ok, thanks. It sure seems like you've narrowed it to the sync but I can't see what. Are you set to download lyrics on playback? Those would get synced back too.
I'm starting to wonder if there is something specifically with your client machine going on. What is your output device?
Can you zip up your settings files (for the client) and email them to me bob (at) jriver (dot) com ?

mwillems · « **Reply #40 on:** February 07, 2020, 09:54:23 am »

Quote from: bob on February 07, 2020, 09:15:46 am

Ok, thanks. It sure seems like you've narrowed it to the sync but I can't see what. Are you set to download lyrics on playback? Those would get synced back too.

I thought that might be it and disabled it, but it made no difference sadly

Quote

I'm starting to wonder if there is something specifically with your client machine going on. What is your output device?

So I can reproduce this (on prior builds) on three different amd64 linux client machines (one debian, two arch), and they all use different output devices, so I think it's more likely something specific to my network environment. But in the interest of troubleshooting: one is outputting to a PCI device (an old asus ST with daughter board), one is outputting to a USB DAC (an ODAC), and the other is outputting to an internal laptop soundcard (integrated in the mobo).

Quote

Can you zip up your settings files (for the client) and email them to me bob (at) jriver (dot) com ?

You mean a library backup? I'll send one from one of the clients shortly.

Hendrik · « **Reply #41 on:** February 08, 2020, 04:38:48 am »

I've sent you a PM with a special version and some instructions, if you're able and willing.

mwillems · « **Reply #42 on:** February 08, 2020, 10:18:39 am »

So, I'm testing now, but it's very slow-going. Valgrind slows playback to a crawl so a five-minute song might take ten or twenty minutes to play. I'll let it run for a few hours and report back.

In terms of methodology, I ran the debug build without valgrind first as a control. I played one album and added another to the queue and let it run for four or five songs. I saw memory growth with the debug build to about 1G and rising. Now I'm running with valgrind using the exact same albums, etc., so hopefully we can capture something.

mwillems · « **Reply #43 on:** February 08, 2020, 11:06:43 am »

Ok I ran it with valgrind per your instructions for a good while. The output is gzipped and attached (there didn't appear to be anything private/personally identifiable in the output). One note is that the memory usage appeared to grow much, much more slowly while running under valgrind (after twelve songs or so the memory never got close to 1GB, that I saw), but the memory usage did appear to be slowly rising even so. Hopefully this helps, and let me know if I can help with anything else!

Hendrik · « **Reply #44 on:** February 08, 2020, 12:00:02 pm »

I'm not seeing any growth in the data. It pretty fast comes up to around 600mb and then stays there pretty consistently - a similar result to what I'm seeing here locally.

mwillems · « **Reply #45 on:** February 08, 2020, 12:09:35 pm »

Quote from: Hendrik on February 08, 2020, 12:00:02 pm

I'm not seeing any growth in the data. It pretty fast comes up to around 600mb and then stays there pretty consistently - a similar result to what I'm seeing here locally.

The reported memory looked like it grew slightly on my end, but I know what you mean. I tried it again without valgrind and it grew plenty fast, so it seems like something about the valgrind environment is changing the behavior. Not sure what that might be though.

Is it possible to run valgrind multi-threaded? It seems to peg one core for thirty seconds or a minute or so every minute or so and the interface and music playback stutter badly until it finishes whatever it's doing (presumably making a checkpoint?), and then runs normally again for a while thirty seconds or so. I wonder if some of the network activity is just timing out and not completing, maybe? If there's an easy way to run it across multiple cores that might let it act more normally.

mwillems · « **Reply #46 on:** February 09, 2020, 10:47:57 am »

So the inability to reproduce the leak in valgrind got me thinking: valgrind is kind of a software virtual machine that executes the code in a processor-neutral virtual CPU, right? So it made me wonder if CPU architecture might have something to do with it. Most of my home build out was done circa 2013-2015, so most of my hardware is haswell or bay trail hardware (e.g. I can repro the memory leak on an i7-4790K, an i5-4250U, and an Atom Z3735F). I don't have much newer hardware floating around, but I wonder if that might be a clue?

Hendrik · « **Reply #47 on:** February 10, 2020, 04:04:14 am »

That seems rather unrealistic. Memory allocation is a software thing.
What puzzles me the most is how this wants to be a Linux-exclusive problem. I've run all sorts of memory debuggers (leak finder, heap profiler, the works) and found no suspect results at all.

An actual leak would show up immediately. And even if it somehow stores memory references somewhere that are only free'ed on exit, those would've shown in the heap profiler.

JimH · « **Reply #48 on:** February 10, 2020, 06:16:25 am »

A driver? Or a plug-in?

bob · « **Reply #49 on:** February 10, 2020, 08:44:06 am »

Perhaps it a system library mismatch issue?

I'll make another test build on buster and we'll see if the issue goes away for you.

INTERACT FORUM

Author Topic: JRiver on Raspberry Pi has a memory leak? (Read 9054 times)