Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[madbrain]

I am using MC 23.0.61 (registered) on an Odroid XU4 single board computer running this OS version :

odroid:~/Desktop> uname -a
Linux odroid 3.10.105-141 #1 SMP PREEMPT Thu Jul 27 04:03:10 UTC 2017 armv7l armv7l armv7l GNU/Linux
odroid:~/Desktop>

I have a 6TB drive attached by USB 3.0 dock, with a 1.1 TB music collection.

I created a new local library, and ran import manually. Import summary showed that it imported 51386 new files.
However, the summary at the bottom of the MediaCenter window said "51386 files - 68.7GB 15.9 days" . It should be closer to 1.1TB - 155 days.
Also, as soon as I clicked done on the "import details", and then clicked somewhere in the audio tab, to "files", or "panes", the summary at the bottom changed to show only about 5000 files. Right now it's showing exactly 5000.
But I have done the import several times, and sometimes it would show up to about 5500.

Any idea what the problem might be ?

I also have a Raspberry Pi and will try to reproduce the problem on it too.

[madbrain]

I also have a Raspberry Pi and will try to reproduce the problem on it too.

Same exact problem on Raspberry Pi. Import folder claims to have imported all the files, but afterwards, it's showing 5001 files only.

[blgentry]

Sounds like MC doesn't recognize your files.  Have you checked all of "Audio" in the import dialog?  Maybe you have some unusual file extension that MC doesn't know?

Brian.

[madbrain]

Sounds like MC doesn't recognize your files.  Have you checked all of "Audio" in the import dialog?  Maybe you have some unusual file extension that MC doesn't know?

Brian.

I had "audio" checked. Media Center 23 recognizes (almost all) the files. Here is a screenshot at the end of the import with the details of the import. See the "import summary" in the lower left, which shows 51,386 files imported. I clicked the "Import details" to see which the 20 files that failed import were. You can see the 51,386 files at the bottom in that shot (but, oddly, still only 68.7 GB / 15.9 days) :



And here is another screenshot immediately after I closed the "import details" dialog and went to "panes" :



As you can see, it reverts back to 5000 files. Seems to be some kind of limit.
This is very easily reproducible on both my single board computers - Odroid XU4 and Raspberry Pi, both ARMv7.

I did not try on Windows PC yet. I can try to create a new library just to test import.

[aoqw76]

The totals at the bottom is not the grand total, it is the total for the selected files in the current view. It is possible that the view you have selected is only a subset of the whole library. Try clicking on “recent albums” to see the lot. If you click on 1 album it is likely to say 10 tracks and 45 mins run time etc.

[blgentry]

I recall there being several issues with WMA files.  Are most of your files WMA?  I don't think copy protected WMA files are supported.  I also remember something strange happening with WMA support after Windows 10 did some update.  I can't remember the details.

Brian.

[madbrain]

The totals at the bottom is not the grand total, it is the total for the selected files in the current view. It is possible that the view you have selected is only a subset of the whole library. Try clicking on “recent albums” to see the lot. If you click on 1 album it is likely to say 10 tracks and 45 mins run time etc.

Yes, I know that, but typing CTRL-A to select all shows a low total as well.

[madbrain]

I recall there being several issues with WMA files.  Are most of your files WMA?  I don't think copy protected WMA files are supported.  I also remember something strange happening with WMA support after Windows 10 did some update.  I can't remember the details.

Brian.

Yes, most are WMA lossless files. They shouldn't be copy-protected.
I converted all my files to this format earlier this year because that's the only lossless format that my cars built-in audio systems (Chevrolet Volt and Bolt) support.
There were problems with the encoding step, but i finally got past it with help from JRiver. There were never any problems playing back the files, though.
All the files still show up in Media Center for Windows. I tried to create a brand new test library on Windows - MC is running import right now.

[madbrain]

No problem with the import of the same files on Windows 10. Seems to be a problem on ARM/Linux.

[blgentry]

Hmm.  I've never used a WMA file with my system (MC for Mac).  I'd expect WMA to be supported on all platforms but it *is* a proprietary MS format so...

I would create a new view called "Everything" to show not only Audio files, but everything else:  Video, Images, Data, etc.  It's possible that MC has imported the files, but with the wrong Media Type.  That happens every now and then.

It's kind of complicated to explain how to do this unfortunately.  Maybe someone else has a better idea to troubleshoot this.

I predict that your Windows import is going to work just fine with all 51k files.

Brian.

[madbrain]

Hmm.  I've never used a WMA file with my system (MC for Mac).  I'd expect WMA to be supported on all platforms but it *is* a proprietary MS format so...

I would create a new view called "Everything" to show not only Audio files, but everything else:  Video, Images, Data, etc.  It's possible that MC has imported the files, but with the wrong Media Type.  That happens every now and then.

I just created a view by "disk location" in the root of MC. It shows 5003 files now with CTRL-A. Not sure why 3 more, but it's still missing a whole bunch.
Anyway, I am using MC for audio only. I would use something other than WMA if my car supported it, but its that or lossy MP3, so I choose WMA.

It's kind of complicated to explain how to do this unfortunately.  Maybe someone else has a better idea to troubleshoot this.

Sounds like a bug if so. This was a simple import with library created from scratch, and completely reproducible.

I predict that your Windows import is going to work just fine with all 51k files.

Yes, it did.

[blgentry]

Customize your disk location view (right click > customize view).  Then do:  Settings > Included Files > use parent scheme rules > (unchecked)

This should show you everything, including "non audio" files.

Brian.

[blgentry]

It's also worth checking:

Customize View > Settings > Included Files > Set rules for file display

Make sure there isn't a rule there to limit by Media Type.

Brian.

[madbrain]

Customize your disk location view (right click > customize view).  Then do:  Settings > Included Files > use parent scheme rules > (unchecked)

This should show you everything, including "non audio" files.

Brian.

Thanks. Unchecked this, but it didn't show any more files.
There was no rule under included files.

I'm still stumped by this.

I can use the PC as media server right now, but it uses 50-100x as much power as either the Odroid XU4 or the Pi, so this is not a long term solution.

[madbrain]

Another test I just did :

On Windows, if I search for "mp3", I find that I have 12,225 MP3 files. I kept them in this format since that's how I got them in the first place, and converting to a lossless format would only waste space. Both car audio systems support MP3 as well.

On Linux on my ODroid, the same search in Media Center finds only 1,542 MP3 files.
find . |grep -i mp3 | wc -l
outputs 12225, so all the mp3 files are there on the Odroid too.

So, I think this confirms that the problem is not specific to WMA files, it seems related to the library size.

I would really appreciate if somebody from JRiver could chime in on this.

[blgentry]

It definitely has nothing to do with library size.  MC has users that have 500,000 or more files in their libraries.

You should try manually importing some of the files that have failed to import and see what happens.

I'm also stumped by this...

Brian.

[madbrain]

It definitely has nothing to do with library size.  MC has users that have 500,000 or more files in their libraries.

You should try manually importing some of the files that have failed to import and see what happens.

I'm also stumped by this...

Brian.

Well, I copied just the 12,225 MP3 files to a separate folder on the Odroid, created another library, and imported just that new folder in MC23 on Linux. All of the MP3 files imported successfully, and still showed up afterwards.

Most of the other 38k files are WMA.

I tried your suggestion. I went back to the original problematic after will import, with only 5003 files showing up.
I then imported a single folder of 28 WMA files that were missing.

The import succeeded. After I dismissed the import dialog, those files were still there, and the library size at the bottom was still 5031 files vs 5003 before.

So, somehow, things go bad when I import the entire tree, but are OK if I import a single folder. I have hundreds of folders with WMA files in them, though. It's not realistic to import them each one at a time. The GUI on the Odroid is abysmally slow, also, which makes all these attempts in MC very difficult. We are talking 4-5 seconds between clicks, easily.
I only want to use the box as a media server and nothing more.

As a file and network server, the box does great, it can stream 600 Mbit/s over Gigabit from a SATA NAS drive connected via USB 3.

[blgentry]

Sounds like that thing is way underpowered.  You might consider a "real" computer instead.

I use a Raspberry Pi 3 as one of my MC systems.  But I don't import files into the Pi directly.  Instead, what I do is this:

1.  Copy all of my music files from my main MC installation to an external drive.  Then I make a Library Backup and store the backup zip file on this same drive.
2.  Mount this drive on the Pi on a specific path like /Volumes/my_drive_name .  Then the Pi can see all of the files.
3.  Using MC on the Pi, I restore the backup zip file that I previously stored on the external drive.  Once that's done, MC sees entries for ALL of my music.
4.  However, MC thinks all of that music is MISSING, because the file paths are wrong.  It's still using the file paths from my main MC system.  So, I select all files and use the Rename, Move, and Copy files tool in UPDATE mode.  I use the Find & Replace template to change the file paths on all of these files to where they now reside on the external drive.
5.  Once I verify that MC can see all of my files again, I rebuild thumbnails.  This is necessary because on-the-fly thumbnail generation can crash MC on the Pi because it can run it out of memory.  Pre-generating all of the thumbnails avoids this.

This procedure keeps me from ever having to import files.  The real benefit is that ALL of my library fields are retained, exactly as I want them, from my main MC computer, to my MC on Pi.

I'm not sure if this procedure would work for you and your workflow, but you might give it a try and see.

Good luck.

Brian.

[madbrain]

Sounds like that thing is way underpowered.  You might consider a "real" computer instead.

I use a Raspberry Pi 3 as one of my MC systems.  But I don't import files into the Pi directly.  Instead, what I do is this:

1.  Copy all of my music files from my main MC installation to an external drive.  Then I make a Library Backup and store the backup zip file on this same drive.
2.  Mount this drive on the Pi on a specific path like /Volumes/my_drive_name .  Then the Pi can see all of the files.
3.  Using MC on the Pi, I restore the backup zip file that I previously stored on the external drive.  Once that's done, MC sees entries for ALL of my music.
4.  However, MC thinks all of that music is MISSING, because the file paths are wrong.  It's still using the file paths from my main MC system.  So, I select all files and use the Rename, Move, and Copy files tool in UPDATE mode.  I use the Find & Replace template to change the file paths on all of these files to where they now reside on the external drive.
5.  Once I verify that MC can see all of my files again, I rebuild thumbnails.  This is necessary because on-the-fly thumbnail generation can crash MC on the Pi because it can run it out of memory.  Pre-generating all of the thumbnails avoids this.

This procedure keeps me from ever having to import files.  The real benefit is that ALL of my library fields are retained, exactly as I want them, from my main MC computer, to my MC on Pi.

I'm not sure if this procedure would work for you and your workflow, but you might give it a try and see.

Good luck.

Brian.

I have a Raspberry Pi 3 also. The GUI is faster on the Pi. But the I/O on the Pi sucks, about 1/5th as fast as the XU4.
XU4 actually is great hardware and much better specs than the Pi3. It's the software that's the weak point due to support not being as good. It doesn't make much sense that the desktop would render so slowly over VNC, but it does. MC UI is faster on the Pi but not usable as a media server.

https://www.slant.co/versus/5727/15521/~odroid-xu4_vs_raspberry-pi-3-model-b

I do use an external SATA drive (actually, SATA drive in a USB 3.1 dock) with my XU4, but I'm using an EXT4 file system as I have found that NTFS was way too slow on the XU4 - the write speed goes down to 5MBytes/s , and makes the XU4 unusable as a file server. On the PI3, the network is the bottleneck, not the drive or filesystem.

I can't mount the EXT4 on my Windows machine .
I can however copy all the files over LAN, and it is fast enough that way. That is what I did.
Some of the files did not copy from NTFS to EXT4 over LAN because of file system differences - might be restrictions on some characters and depth of directories, not sure.
But 99% of the files did copy.

I will try your method, but I feel like local file import should work. The XU4 has plenty fast I/O and it takes <20 min to scan all 50k files, fairly reasonable. Problem is that they don't actually show up in the library afterwards.

[blgentry]

Something is clearly wonky with your setup.  I think MC is the least likely problem, but I could certainly be wrong.  It might be MC's fault.  I suspect your hardware because you say it is so slow.  Again, I could be wrong.

What would really tell the story is if MC on a "real" Linux PC imported all of your files successfully.  If that failed also, the MC for Linux would be a real suspect in this import problem.

BTW, this is a separate debate, but MC on the Pi works just fine as a media player.  I control mine with EOS remote on Android and it does everything I want.  Maybe you have some sort of different use case from me.

Brian.

[madbrain]

Something is clearly wonky with your setup.  I think MC is the least likely problem, but I could certainly be wrong.  It might be MC's fault.  I suspect your hardware because you say it is so slow.  Again, I could be wrong.

The XU4 is extremely fast when used from an SSH terminal. Only the GUI is slow. I haven't bothered to try to solve it since I am using the box mostly as a file/media server.

What would really tell the story is if MC on a "real" Linux PC imported all of your files successfully.  If that failed also, the MC for Linux would be a real suspect in this import problem.

I don't have a "real" Linux PC, but I could try a VM.

BTW, this is a separate debate, but MC on the Pi works just fine as a media player.  I control mine with EOS remote on Android and it does everything I want.  Maybe you have some sort of different use case from me.

Not sure if you noticed, but I tested MC on my Pi3 also, and it has the exact same problem as the XU4 importing my library and showing only about 5000 files afterwards.
It's definitely not an issue just with my XU4 hardware, since it's common to both the XU4 and Pi3.

I would say it's likely a bug with the MC Linux ARM version that is only triggered in a specific situation with my library.
Not sure if it exists on MC Linux x86/x64 too, but that's not the platform I'm using.

And yes, our use cases are quite different. My Pi3 and XU4 are both running headless. The only things attached are network cable, Ethernet cable, and some USB hard drives.
I need fast I/O for movies - my BR movie collection is in ISO on the hard drive. Pi3 can barely stream one Bluray ISO with its networking. It cannot stream 2 Bluray ISO to different clients in the house.

I have also added 13 Chromecast audios in the house for many different locations. Big house. Will be adding 2 more CA this week. This needs network bandwidth too when using lossless and streaming to the Chromecast audio losslessly. Epecially if one is playing different stuff in each zone. Pi3 I/O simply cannot do it. XU4 can. Only problem is MC cannot deal with my library on either one so far ...

[blgentry]

I don't have a "real" Linux PC, but I could try a VM.

That might tell you something.

Not sure if you noticed, but I tested MC on my Pi3 also, and it has the exact same problem as the XU4 importing my library and showing only about 5000 files afterwards.

Ahhh, right.  I think I glossed over that detail.  Seems like my theory about your hardware is probably incorrect.

I would say it's likely a bug with the MC Linux ARM version that is only triggered in a specific situation with my library.

You talked above about copying folders to different folder names.  Then you talked about importing a problematic folder of 28 WMAs.  Did you copy that folder somewhere else too?  I ask because I wonder if there are some "strange" characters in one or more of your top level directories that might be confusing the import process in some way.

And yes, our use cases are quite different. My Pi3 and XU4 are both running headless. The only things attached are network cable, Ethernet cable, and some USB hard drives.

My Pi3 is the same:  Headless.  Not even an ethernet cable actually; just wifi, power cable, USB hard drive, and USB DAC.

I need fast I/O for movies - my BR movie collection is in ISO on the hard drive. Pi3 can barely stream one Bluray ISO with its networking. It cannot stream 2 Bluray ISO to different clients in the house.

Hmmm.  I'd be skeptical of that working well.  But hey, that's not the issue we are trying to solve so...

I have also added 13 Chromecast audios in the house for many different locations. Big house. Will be adding 2 more CA this week. This needs network bandwidth too when using lossless and streaming to the Chromecast audio losslessly. Epecially if one is playing different stuff in each zone. Pi3 I/O simply cannot do it. XU4 can. Only problem is MC cannot deal with my library on either one so far ...

I understand that you are explaining how fast the networking is on your XU4.  I'm unconcerned with that for purposes of this discussion, but it's interesting to know; might be useful to me in the future.

I guess I really don't have anything more to suggest at this point.  Maybe one of the staff here could figure something out if you enabled logging, did the import, and then attached the log file here.

Brian.

[madbrain]

You talked above about copying folders to different folder names.  Then you talked about importing a problematic folder of 28 WMAs.  Did you copy that folder somewhere else too?  I ask because I wonder if there are some "strange" characters in one or more of your top level directories that might be confusing the import process in some way.

No, I didn't copy that WMA folder anywhere else. It was just a subfolder from the original tree that I had imported in full.

My Pi3 is the same:  Headless.  Not even an ethernet cable actually; just wifi, power cable, USB hard drive, and USB DAC.

My Pi3 and XU4 are in a home office. No DAC attached. The real PC next to it has 2 Firewire audio interfaces, Echo Audiofire 12 and 8a, total of 20 analog in / 20 analog out, with great 192 kHz/24 bit DACs and ADCs on each one (though the Windows driver unfortunately only supports 96 kHz, Mac driver does 192k but I don't have one).

Hmmm.  I'd be skeptical of that working well.  But hey, that's not the issue we are trying to solve so...

There really is nothing to solve there, that part just works. I just mount share the XU4 drive by Samba, mount it on the HTPC, double click on the ISO to mount it with Virtual clone drive, and play with PowerDVD. Have watched many BR movies that way - no problem. One HTPC downstairs is wired gigabit, the other HTPC upstairs is on Wifi AC with about 400 Mbps effective throughput.
The Toshiba N300 6TB drive I bought recently can easily handle 2 BR streams, and the XU4 networking keeps up, as does the Orbi Wifi mesh.

I guess I really don't have anything more to suggest at this point.  Maybe one of the staff here could figure something out if you enabled logging, did the import, and then attached the log file here.

Sure.

[blgentry]

My Pi3 and XU4 are in a home office. No DAC attached. The real PC next to it has 2 Firewire audio interfaces, Echo Audiofire 12 and 8a, total of 20 analog in / 20 analog out, with great 192 kHz/24 bit DACs and ADCs on each one (though the Windows driver unfortunately only supports 96 kHz, Mac driver does 192k but I don't have one).

OK I'm way confused.  Why are you using the Pi and XU4 at all if you have a real PC with good DACs?  Oh wait, for the Chromecasts?  Why even bother?  Why not use the PC?

Brian.

[madbrain]

OK I'm way confused.  Why are you using the Pi and XU4 at all if you have a real PC with good DACs?  Oh wait, for the Chromecasts?  Why even bother?  Why not use the PC?

Brian.

The real PC and the great DACs are used for recording, mainly. And they are only in a single room.
The real PC uses tons of power and can't be used 24/7 as a server for that reason. Everything is always in standby mode when not in use. Amps/DAC/etc are all on remote X10 switches to turn them off when not in use. Everything that's idle and consuming significant power also generates heat, which in turns drives up AC costs too. The price of electricity is expensive in California. Even though I have solar PV, I also have 2 electric cars also, and a huge house, with 2 central ACs, a hot tub, and a sauna, I still end up having net positive usage, and a net bill. Marginal cost is still about 30 cents/kWh. A 100W PC (that's about how much it uses when idle) will cost $262/year to run 24/7. An XU4 at 20W (counting external SATA dock and drive) will be $52/year in comparison.  So, that is the rationale.

I bought the Pi and XU4 exclusively to limit the power consumption, and have 24/7 access to media / files from anywhere in the house (smartphone, HTPCs, and now Chromecsts). I like that they are fanless and quiet, also, vs the PC which isn't.

All the Chromecasts were added just recently in the last week to finally make better use of all the ceiling speakers that came with the house. Just about 15W idle power consumption for a total of 13 Chromecasts. And I may put those on remote power switches too since the 5 amps needed to power 13 zones consume a total of 300W idling, and definitely have to be on switches. 300W running 24/7 = enough electricity to do all the driving in a year for one of my two EVs. My next project is to play with the Wifi smartplugs I bought this weekend for this purpose. X10 PLC signals won't work well accross floors and half the amps and Chromecast are upstairs, half downstairs. I'm afraid I have to retire X10 in my house, at least for new usage. I have 4 X10 hardwired light switches in my house, all on the first floor, and PLC signals sent by my XU4 or Pi3 reach all of them, but won't be adding more X10 gear to my house.

[bob]

Not seeing that here.
I just imported 9700 files at one time from a NAS share.
You might turn on logging...

Also note that you can create a smartlist with deleted and bad entries in it then view the smartlist to see if there are entries in there for your "missing" files.