Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[mwillems]

So I've been struggling with my pi jriver builds lately, and after spending a few days troubleshooting, I think I've at least got a handle on what's happening, although I'm not sure why. 

Here's the basic observed behavior: jriver starts on the pi and slowly adds memory until it sits at around 40% of the pis available memory at idle.  Then if I play music, each track increases the total memory consumed by jriver, but previous track memory is not fully relinquished which leads to a gradually increasing memory footprint until jriver exhausts available memory at which point things start swapping leading to the OOM killer killing the MC process after a few minutes.  I can reproduce this just by playing music to it for a half an hour or an hour; at times it can't even finish an album without crashing. 

I'm not sure when this behavior started as I had been having pi hardware problems for a month or so (I replaced the defective hardware), but it was definitely not the case earlier in the MC 23 dev cycle.  Stopping playback releases a little memory, but not all of it (i.e. it might go from 80% of system memory back down to 72%, but not back down to the baseline 40%).  It doesn't seem to make a difference how playback is initiated: I see the same result starting playback on the GUI on the pi, using gizmo, or starting playback fromt the server.  It also doesn't seem to make a difference whether the "buffer to disk" options are checked or unchecked as well.

Contextual info:
1) All thumbnails are built.
2) The pi is a client to a library server
3) This occurs with only the media server and the pi on the network.
4) The media server is running windows 10.
5) The DLNA Server and Controller options are unticked on the pi.
6) The only external hardware attached to the pi is a hifiberry amp+, but it uses the same board and driver interface as the Hifiberry DAC+, which I think has been pretty well-tested.
7) MC version is 23.0.102 (which I think is the latest arm version)

Let me know if logs would be helpful, or if anyone has seen anything similar you can offer tips on.

[mwillems]

Ok, I did a bunch more differential testing, and I think I've found the culprit:  if I untick the box labelled "Auto Sync with Server" under Media Network-->Client Options on the pi, the memory usage levels off and stays level.  If I tick the box, I get the behavior I described above, as well as slowly increasing memory usage even when nothing is playing (which I hadn't spotted previously because it's slow).  Looking at things in netstat it looks like there are tcp sockets that never close and just kind of keep accumulating, which might account for the increased memory usage? 

Not sure why, but that setting appears to be the issue.  I previously had that setting disabled, so the leak is not likely to be new (i.e. it may have existed indefinitely, I would have never seen it).  I can actually reproduce this on the amd64 build too now on my Arch Linux machine, so I think that setting may just cause a memory leak (at least when a linux client connects to windows server).

[bob]

Ok, I did a bunch more differential testing, and I think I've found the culprit:  if I untick the box labelled "Auto Sync with Server" under Media Network-->Client Options on the pi, the memory usage levels off and stays level.  If I tick the box, I get the behavior I described above, as well as slowly increasing memory usage even when nothing is playing (which I hadn't spotted previously because it's slow).  Looking at things in netstat it looks like there are tcp sockets that never close and just kind of keep accumulating, which might account for the increased memory usage? 

Not sure why, but that setting appears to be the issue.  I previously had that setting disabled, so the leak is not likely to be new (i.e. it may have existed indefinitely, I would have never seen it).  I can actually reproduce this on the amd64 build too now on my Arch Linux machine, so I think that setting may just cause a memory leak (at least when a linux client connects to windows server).

Interesting.
I wonder if it's an issue in the Mac and Windows builds as well.

[mwillems]

If it exists on windows its either newish or very slow as I had windows clients with that setting enabled running for literally months about a year ago (before I migrated my jriver server back to windows).  I had noticed weird leaky behavior on my amd64 Arch machine for more than a year, but I don't always use that machine as a client, so I could never pin it down.  It's nice to at least get a differential diagnosis going :-)

If you can't repro it let me know, and I can try to provide more info.

[bob]

If it exists on windows its either newish or very slow as I had windows clients with that setting enabled running for literally months about a year ago (before I migrated my jriver server back to windows).  I had noticed weird leaky behavior on my amd64 Arch machine for more than a year, but I don't always use that machine as a client, so I could never pin it down.  It's nice to at least get a differential diagnosis going :-)

If you can't repro it let me know, and I can try to provide more info.

Trying to repro it.
Did you use lsof to check for the hung sockets?
Are you using any of the other client options like audio transcoding?

[mwillems]

Trying to repro it.
Did you use lsof to check for the hung sockets?
Are you using any of the other client options like audio transcoding?

I tried it both with and without transcoding.  Orignal files were FLAC and transcoding to medium quality mp3 seemed to slow the leak somewhat, but otherwise had no effect.  I used netstat -p tcp to observe the sockets, I'll try lsof when testing tomorrow.  It's not a subtle leak during playback, on my amd64 system which has 16Gb of ram, MC will be using 30 or 40% of the ram after three or four hours of playback.

[mwillems]

So testing on my main (non-pi) machine. I'm not seeing the hung sockets (that may have just been an artifact of the pi's lame-o networking), but the memory leak is definitely there and turns on or off depending on the auto sync with server setting.  The memory leak seems to increase by (more or less) the size of the file played; every file played increases the memory usage by roughly its size.  There's also the slow leak over time too even at idle.  One odd thing I noticed which may or may not be a clue: when I tried toggling auto-sync with server from on to off during playback I saw really odd behavior.  The currently playing song kept playing, but the stop button wouldn't stop it (even though the MC UI showed the song as stopped).  Playing another song resulted in both playing at once and the UI successfully controlled playback for the second song only.  Weird. 

Here's a diff of the lsofs in the two states during playback (autosync on is the first argument to diff, autosync off is the second argument to diff):

Code: [Select]

   COMMAND     PID    USER   FD      TYPE             DEVICE  SIZE/OFF     NODE NAME

< mediacent 10278 michael  DEL       REG                0,5             111650 /memfd:pulseaudio
---
> mediacent 10278 michael  DEL       REG                0,5             110720 /memfd:pulseaudio
7c7
< mediacent 10278 michael  DEL       REG                0,5              57022 /memfd:pulseaudio
---
> mediacent 10278 michael  DEL       REG                0,5             108267 /memfd:pulseaudio
139c139
< mediacent 10278 michael   19u     IPv4             109974       0t0      TCP *:52199 (LISTEN)
---
> mediacent 10278 michael   19u     IPv4              57139       0t0      TCP *:52199 (LISTEN)
140a141
> mediacent 10278 michael   21u     IPv4             110672       0t0      TCP Neptunian:52199->living_room.system:57196 (ESTABLISHED)
142d142
< mediacent 10278 michael   23u      REG              259,6  23664205  3145755 /home/michael/.jriver/Media Center 23/Temp/BufferedInternetReader - 2663028480-0.dat.cnk
143a144
> mediacent 10278 michael   28u      REG              259,6  18722850  3145755 /home/michael/.jriver/Media Center 23/Temp/BufferedInternetReader - 3437180672-0.dat.cnk
147,151c148,152
< mediacent 10278 michael   32r      REG              259,6  23664205  3145755 /home/michael/.jriver/Media Center 23/Temp/BufferedInternetReader - 2663028480-0.dat.cnk
< mediacent 10278 michael   33r     FIFO               0,12       0t0    57019 pipe
< mediacent 10278 michael   34w     FIFO               0,12       0t0    57019 pipe
< mediacent 10278 michael   35r     FIFO               0,12       0t0    57020 pipe
< mediacent 10278 michael   36w     FIFO               0,12       0t0    57020 pipe
---
> mediacent 10278 michael   32r      REG              259,6  18722850  3145755 /home/michael/.jriver/Media Center 23/Temp/BufferedInternetReader - 3437180672-0.dat.cnk
> mediacent 10278 michael   33r     FIFO               0,12       0t0   108264 pipe
> mediacent 10278 michael   34w     FIFO               0,12       0t0   108264 pipe
> mediacent 10278 michael   35r     FIFO               0,12       0t0   108265 pipe
> mediacent 10278 michael   36w     FIFO               0,12       0t0   108265 pipe
153c154
< mediacent 10278 michael   38u     unix 0x00000000926fb9fb       0t0    57023 type=STREAM
---
> mediacent 10278 michael   38u     unix 0x00000000ba652056       0t0   108268 type=STREAM

Of note: turning auto-sync off stops the growth in memory usage, but does not release any memory previously built up with auto-sync on, and I took the lsof of auto-sync off after auto-sync on, so the total memory usage was larger than when I took the auto-sync on lsof snapshot.  Let me know if I can provide any other useful info (or clean lsofs right after opening MC or something).

[craigmcg]

Quote from Bob

"Interesting. I wonder if it's an issue in the Mac and Windows builds as well."

I can't say that it is for sure but when I turned off Auto synch on the client PCs (all running Win 10 x64), the library server on the server PC stopped the crashing that has been bugging me for months (first thought that it was network switch related then Win 10 networking changes, etc.). While I only made the change yesterday I haven't had the crash on the server PC since (was happening a few times each day). I have 5-6 client PCs connecting to the server using the Library server.

Thanks to Bob and mwillems for pointing me to what seems to be the solution.

[mattkhan]

FWIW I have 2 separate linux machines connected to a windows server, I do have that option checked and I don't see any evidence of a socket leak

[bob]

Are any of you using Last.FM updating?

[craigmcg]

I'm not using Last FM.

[mwillems]

Are any of you using Last.FM updating?

I'm not using last.fm either.

[mwillems]

FWIW I have 2 separate linux machines connected to a windows server, I do have that option checked and I don't see any evidence of a socket leak

Do either of your clients have media network enabled?  Or are they just connected as clients to the library server?  Just trying to think of other config differences that might be relevant.  In my case media network is enabled on all computers, but DLNA controller and server are disabled on the clients.

[mattkhan]

  In my case media network is enabled on all computers, but DLNA controller and server are disabled on the clients.

I have the same setup though my Linux boxes run Debian Buster

[bob]

Quote from Bob

"Interesting. I wonder if it's an issue in the Mac and Windows builds as well."

I can't say that it is for sure but when I turned off Auto synch on the client PCs (all running Win 10 x64), the library server on the server PC stopped the crashing that has been bugging me for months (first thought that it was network switch related then Win 10 networking changes, etc.). While I only made the change yesterday I haven't had the crash on the server PC since (was happening a few times each day). I have 5-6 client PCs connecting to the server using the Library server.

Thanks to Bob and mwillems for pointing me to what seems to be the solution.

Is your system all windows (the server PC is windows 10 as well)?

[craigmcg]

Yes they are all on Windows. This said, I did have one laptop on Linux at one laptop running as a client a few weeks ago. I have to wipe reload a PC so I can load it  with Linux if this would be helpful.

[bob]

Yes they are all on Windows. This said, I did have one laptop on Linux at one laptop running as a client a few weeks ago. I have to wipe reload a PC so I can load it  with Linux if this would be helpful.

The linux machine shouldn't be able to crash the server on Windows so the fact that the windows MC is crashing from having that switch on is good information.
Any chance you could switch back to that temporarily to get a crash dump file?

[craigmcg]

Just to be clear, the Library server (on Windows) was crashing when clients (no Linux running) had Auto synch on (and probably was with a Linux client laptop but not sure). Do you need me to do a Linux install with the Auto synch on, just re-enable Auto synch on one (or more) Windows clients, or both before enabling logging on the Library server to get a crash dump? I'm assuming that it would be advantageous to be able to isolate whether or not the system crashes with only one  OS involved.

[bob]

Just to be clear, the Library server (on Windows) was crashing when clients (no Linux running) had Auto synch on (and probably was with a Linux client laptop but not sure). Do you need me to do a Linux install with the Auto synch on, just re-enable Auto synch on one (or more) Windows clients, or both before enabling logging on the Library server to get a crash dump? I'm assuming that it would be advantageous to be able to isolate whether or not the system crashes with only one  OS involved.

The crash dump from the windows machine would be most useful to start with. I assume this will take a few steps to track down the issue on both ends. Thanks.

[craigmcg]

So I enabled logging on the Library server PC then reactivated auto synch on the clients two days ago and no crashes yet. Sigh. I'll send the dump if it does crash.

Craig