Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Mastiff]

I have MC set up on a Pi 4 with 4 gig RAM for my boat, mostly headless and controlled by JRemote, but I log in with VNC to do updates of the library, import new files and stuff like that. My problem is that when I have it in Playing now (I have it starting up there to minimize the number of files it has to check the disk for at startup) and then click on Audio, it takes up to 4-5 minutes before anything happens, often I have to click a few times too. I have a library of around 175 000 tracks, so I'm guessing that's the reason. But is there anything I can do to avoid these "lockups" (which aren't really lockups because something happens sooner or later)?

[stefano_mbp]

That amount of tracks is very huge... are you using an hdd or ssd? Maybe an ssd could help to make the system faster

[Mastiff]

I have a 2 TB USB3 disk, and an SSD that size is too expensive for this project, I'm afraid. From 400 Euro and up to 500. Anyway, the disk is quite fast for a mechanical drive.

[stefano_mbp]

Check on Amazon, actually they are around $200, Crucial and Sandisk just below $200 ... not that much ...

[Mastiff]

When you add shipping to Norway and Norwegian VAT it won't be that much different.

[stefano_mbp]

.... beautiful land ... but so expensive ....

[Mastiff]

Very. But we do have great black metal! Listening to Enslaved's new "Utgard" right now.

[stefano_mbp]

...I love Grieg’s Peer Gynt, the full version is really exciting...

[Mastiff]

Very early black metal. He was pretty heavy for his time.

[JimH]

I have MC set up on a Pi 4 with 4 gig RAM for my boat, mostly headless and controlled by JRemote, but I log in with VNC to do updates of the library, import new files and stuff like that. My problem is that when I have it in Playing now (I have it starting up there to minimize the number of files it has to check the disk for at startup) and then click on Audio, it takes up to 4-5 minutes before anything happens, often I have to click a few times too. I have a library of around 175 000 tracks, so I'm guessing that's the reason. But is there anything I can do to avoid these "lockups" (which aren't really lockups because something happens sooner or later)?

It would be worth trying something other than VNC.

Any pictures of your boat?

[Mastiff]

Actually I was a bit imprecise there. The same thing happened when it was connected to a monitor and a keyboard while installing. Right after a startup it seems OK, but give it a few hours, and it seems like MC somehow needs a few minutet wakeuptime.

And of course I do! Thank you for asking!  This is the Mad Max loaded for lobster fishing. Note the boxes with 6x9 JBL marine speakers under the roof. There's another pair under the roof inside the wheelhouse, behind the bench. And there's a 10" JBL marine subwoofer in the box in front of the wheelhouse. The Pi and the disk are in a IP65 fuse box inside the wheelhouse, and I have it input on aux on a Pioneer car audio headunit, and two Alpine amps drive the speakers.

This video is me Thursday, on my way out to set the first lobster pots, with Alestorm's "Tortuga" blasting. Pirate themed folk metal fits the boat very well, I think!  I didn't know how to embed that...

https://youtu.be/BBkpWTbwtio

[mwillems]

So one tip from my own pi experience: pre-generate all the thumbnails.  I noticed when I was running MC with a Pi3 and a library of ~100,000 files that the library views were many times more responsive when I pre-generated all the thumbs using the "build missing thumbnails" option.  Make sure you've set it to the lowest thumbnailing priority before you do it though (although it should be there by default on a pi).

Obviously your actual media is on the spinning disk, but is your library/database on the spinning drive too or is it on the pi's sd card?  It might be worth trying a different location (move your library/database from spinning drive to sd or vice versa) to see if that makes a difference.

Another thing that I found helped responsiveness on my headless pi systems was restarting MC once a day in the middle of the night with a cronjob.  Not sure why it helped, but it really did.

If you've already tried all that, it might be worth doing some profiling.  For example, if you run the top command in the terminal and then do something that hangs MC.  Then you can see where the bottleneck is (i.e. is a ton of CPU just spinning away in iowait or is MC actually doing something?). 

[Mastiff]

Thanks, mwillems! I have the library on the Micro SD-card that also has the OS. I believe the card reader on the Pi 4 is USB 3.0, and I have a quite fast Sandisk card that I used to have in a DJI Phantom Pro drone for 4K video. But I will try to do that pregeneration of thumnails, I can let it work over night. And I am guessing that the restart of MC (which I will probably implement as a full restart of the Pi with Node-RED) fixes the same thing that I wrote about, the "lockup" after some hours.

I have a CPU graph in the menu line, and I don't think I've ever seen it go above 40 % on the Pi 4. On the Pi 3B I think it went all the way up, if I'm not remembering that incorrectly.

Edit: Come to think about it, I think I have all the covers in a separate directory on the hard drive. Maybe that has something to do with it? I'll try to move that to the Micro SD card. It's a 64 gig card, so I can create a separate partition for that.

[JimH]

Nice boat, Tor!  Thanks for the pictures.  I started getting seasick watching the video.  Nice to have a wheelhouse.

[Mastiff]

Thanks, Jim! And that's actually not bad for this time of year here. Today I had around 2 meter waves...

[JimH]

Thanks, Jim! And that's actually not bad for this time of year here. Today I had around 2 meter waves...

I've done about 1500 miles of open ocean sailing over the years, mostly on the North Sea.  Farsund ring a bell?

I fully appreciate how different it can be.

Thanks again for the pictures!

[Mastiff]

Farsund is about 45 minutes drive west of Mandal, where I live. If I had known you were there (and if it was in a time when I knew who you are...) I'd taken the trip to buy you a beer!

[Mastiff]

Btw if that matters, I have tried running some intense stuff now (like library updates, clear library, mass tagging and so on), and the max CPU usage I see in the graph is just under 40 %. Shouldn't MC use more then that when it's available?

[Awesome Donkey]

They might be single threaded tasks.

[mwillems]

Btw if that matters, I have tried running some intense stuff now (like library updates, clear library, mass tagging and so on), and the max CPU usage I see in the graph is just under 40 %. Shouldn't MC use more then that when it's available?

It's possible the tasks aren't multithreaded, but it's hard to tell what's going on without doing profiling (with tools like top or iotop).  A simple cpu graph won't tell you very much.

For example, a good way to get a sense of the "threadedness" of the work load is the "load" count (get by running "uptime" or looking at the top right of the display in the "top" command).  It tells you how many jobs are running or waiting for a processor at any given moment.  If the load count is higher than the number of processors you have, but you still have low CPU utilization, that tells you that the bottleneck is somewhere else.

Normally the bottleneck on a Pi is I/O not CPU.  If you're not seeing max CPU even on intense tasks its most likely that the CPU simply can't get data as fast as it can process it.  IF you run top and look at the third row, you can see the percentage of CPU usage in various activities.  "wa" is waiting for I/O.  If you see a high percentage there while running an intense task that will answer your question.  Another useful tool is "iotop", which can give you a sense of how much throughput is going on.  You'll probably need to install iotop though as its not there by default on debian like top and uptime.

[Mastiff]

This stuff is a bit out of my league, I'm a Windows man. But the load average is 2,28, 2,24 and 2,18 on a Pi 4. And MC regularly goes over  100 % CPU, and that's just illogical to me! It jumpes from 27 % to 125 %, and up and down. At least memory is no problem, that stays around 9 %.

But I found something else that may explain much of the slowdown. My library has been multiplied when converting it from my main media server library to the Pi! I ran Rename, move and copy files from Library tools with "Update database to point to new location (no file rename, move or copy)" to replace E:\MP3 with /media/pi/MP3 and "Convert Windows File Path Syntax to Mac/Linux" ticked off. Turns out that it didn't work correctly, and the E:\MP3 was replicated with "invalid path" or something like that, while at the same time I had an auto-import set up that I didn't even remember I had, so the library was doubled, to more than 350 000 tracks... Suddenly it got a lot more responsive! 

[mwillems]

This stuff is a bit out of my league, I'm a Windows man. But the load average is 2,28, 2,24 and 2,18 on a Pi 4. And MC regularly goes over  100 % CPU, and that's just illogical to me! It jumpes from 27 % to 125 %, and up and down. At least memory is no problem, that stays around 9 %.

Linux reports CPU utilization on a per core basis, so 125% CPU means one core is fully utilized and another core is 25% utilized.  A load average of 2-ish under load means that there are typically only two processes running or waiting for CPU time on average, which (given that the OS is also running necessary tasks too) suggests that whatever MC is doing is either a series of single-threaded tasks or tasks that are very lightly multithreaded (i.e. don't even use two cores fully).

   
But I found something else that may explain much of the slowdown. My library has been multiplied when converting it from my main media server library to the Pi! I ran Rename, move and copy files from Library tools with "Update database to point to new location (no file rename, move or copy)" to replace E:\MP3 with /media/pi/MP3 and "Convert Windows File Path Syntax to Mac/Linux" ticked off. Turns out that it didn't work correctly, and the E:\MP3 was replicated with "invalid path" or something like that, while at the same time I had an auto-import set up that I didn't even remember I had, so the library was doubled, to more than 350 000 tracks... Suddenly it got a lot more responsive! 

Hey that's great news that you found something out of it!  As I recall MC's process for checking if files in the library are missing is very "expensive" in terms of I/O and CPU, and there's a menu option to disable checking it in views (under Tree & View-->Advanced and called "display missing file image in lists (slow on network drives)").  I suspect that was a big part of your slowdown (i.e. not just doubled library, but also lots of missing items).  You should try disabling that setting; auto import would still clean up if you have fix broken links on, but your views would probably be much faster.

[Mastiff]

Aha, I see. So that's not so bad to go over 100 % on the CPU, then.

And I found that missing files to, under "Options", "Tree & View", "Advanced" there's "Display missing file images in lists". So I have remove that now. I hope that will take care of all my problems with this. Thanks again for the help!