INTERACT FORUM

Please login or register.

Login with username, password and session length
Advanced search  
Pages: [1]   Go Down

Author Topic: Id File Structure and Library Migration  (Read 2311 times)

Shasta Mike

  • World Citizen
  • ***
  • Posts: 219
Id File Structure and Library Migration
« on: June 03, 2022, 11:41:49 am »

Apologies for including many issues - thinking they are possibly related.  Any advice on anything is greatly appreciated. 

In setting up a local library on the Id, I am seeing that the music files seem to have propagated into a second location.  Or, is this the same location with two paths? 
Path 1: /home/media/Music
Path 2: /home/media/Music/ournas/Music

To play from the local library I had to use Path 2.  The number of music files is large and if in two places, could cause the Id to run out of drive space which would be a big problem.  Somewhat answered my own question by checking how much space is being used on the Id.  My best guess is that files are not being duped on the Id but something is creating alternate file paths which I need to understand in order to better deal with this next problem.

Using the Library - Rename function on getting the filepaths to work on the Id from a restored PC library I discovered another possible issue.  I was renaming the path using Find and Replace and don't understand the ramifications of using Update Database to point to new location vs Rename (moves files if directory changes). I started the library update using the Rename choice and then MC started crashing after making a few thousand changes.  So, I changed to using Update Database and MC made the rest of the changes and seems to be working fine. The library has also gone through a couple massive import sessions with the last one taking 10+ hours and 230,000 Thumbnails updated.  I don't know what is causing this either - the first time I understand because of all the changes with Renaming. Windows says there 46,000 files and I know 95% plus of the folders only have one jpeg and maybe an average of seven songs so this doesn't make sense.  There are probably 7-8K cover art files - what would cause the Id MC to update 230K?

Another issue with the Id library is with Cover Art. A stack icon is showing up on a few files and I have not used and didn't even know about stacks until researching this issue.  For example, on one album, there was cover art for the two files with stack icons and not the rest.  When I did a Quick Find for the whole album, I got cover art for the other files and two songs with the stack icon had their cover art disappear and I cannot get it back.  I am not using Stacks and would be happy for it to go away but see no way to do that or know if it serving another purpose I don't understand.  And, one post on stacks from years ago mentioned it can be a data hog which may explain why MC now seems sort of doggy.   In trying to figure out the Cover Art problem I noticed that MC was pointing at the /home/media/Music directory when I checked in Tags.  By doing a Find and replace and pointing to /home/media/Music/ournas/Music that fixed the Cover Art displaying correctly.  I don't understand why Import didn't put these two recently added folders in the /home/media/Music/ournas/Music directory, why they still played OK, and what is going on with these two directories on the Id.  I didn't create the ournas/Music directory and am confused about the behavior - ournas is not a term I have ever used. 
 
Again, would what I did using the Rename cause extra files to be created?  And, does the fact I used two different ways to change the directory cause me problems down the road.  I've learned so much I am tempted to wipe the Id and start over to do it right, but, if it really doesn't matter that would still be a lot of work. 

Hopefully, someone can help me understand these issues better.  If you read this whole missive, ANY feedback and hints appreciated.  Thanks!
Logged

bob

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 13452
Re: Id File Structure and Library Migration
« Reply #1 on: June 06, 2022, 12:05:07 pm »

Apologies in advance for including many issues - thinking they are possibly related.  Any advice on anything is greatly appreciated. 

In setting up a local library on the Id, I am seeing that the music files seem to have propagated into a second location.  Or, is this the same location with two paths? 
Path 1: /home/media/Music
Path 2: /home/media/Music/ournas/Music

We don't encourage altering any of the paths setup.
There shouldn't be anything under the ~media/Music directory except ournas.
ournas is a symlink to the media partition where the all of the media on the Id is expected to be.
The actual path for that on disk is
/export/media
/export is the media partition and media is the writable folder under that.

When you connect to the Id with samba/cifs you will see 2 shares.
Internal Storage which is the /export partition
and
External Storage which is where any devices mounted on the Id will appear (external drives, USB ..)

The default for MC on the Id is to import from /home/media/Music. You shouldn't add any more import paths to MC unless you add an external device to the Id.

Quote
To play from the local library I had to use Path 2.  The number of music files is large and if in two places, could cause the Id to run out of drive space which would be a big problem.  Somewhat answered my own question by checking how much space is being used on the Id.  My best guess is that files are not being duped on the Id but something is creating alternate file paths which I need to understand in order to better deal with this next problem.

Using the Library - Rename function on getting the filepaths to work on the Id from a restored PC library I discovered another possible issue.  I was renaming the path using Find and Replace and don't understand the ramifications of using Update Database to point to new location vs Rename (moves files if directory changes). I started the library update using the Rename choice and then MC started crashing after making a few thousand changes.  So, I changed to using Update Database and MC made the rest of the changes and seems to be working fine. The library has also gone through a couple massive import sessions with the last one taking 10+ hours and 230,000 Thumbnails updated.  I don't know what is causing this either - the first time I understand because of all the changes with Renaming. Windows says there 46,000 files and I know 95% plus of the folders only have one jpeg and maybe an average of seven songs so this doesn't make sense.  There are probably 7-8K cover art files - what would cause the Id MC to update 230K?

Another issue with the Id library is with Cover Art. A stack icon is showing up on a few files and I have not used and didn't even know about stacks until researching this issue.  For example, on one album, there was cover art for the two files with stack icons and not the rest.  When I did a Quick Find for the whole album, I got cover art for the other files and two songs with the stack icon had their cover art disappear and I cannot get it back.  I am not using Stacks and would be happy for it to go away but see no way to do that or know if it serving another purpose I don't understand.  And, one post on stacks from years ago mentioned it can be a data hog which may explain why MC now seems sort of doggy.   In trying to figure out the Cover Art problem I noticed that MC was pointing at the /home/media/Music directory when I checked in Tags.  By doing a Find and replace and pointing to /home/media/Music/ournas/Music that fixed the Cover Art displaying correctly.  I don't understand why Import didn't put these two recently added folders in the /home/media/Music/ournas/Music directory, why they still played OK, and what is going on with these two directories on the Id.  I didn't create the ournas/Music directory and am confused about the behavior - ournas is not a term I have ever used. 
 
Again, would what I did using the Rename cause extra files to be created?  And, does the fact I used two different ways to change the directory cause me problems down the road.  I've learned so much I am tempted to wipe the Id and start over to do it right, but, if it really doesn't matter that would still be a lot of work. 

Hopefully, someone can help me understand these issues better.  If you read this whole missive, ANY feedback and hints appreciated.  Thanks!
I am not an expert on the move/rename functions so I'm not sure how to answer that question. I'd be careful with that, you don't want two copies and you especially don't want things in the /home/media/Music folder that aren't under ournas or you will be filling the limited OS filesystem space.

As for cover art I store all mine in tags for files that support it and for ones that don't (like SACD) I store them in Folder.jpg so they travel with the media.

You might want to try using the portable library function bringing over the windows library.
The prefix for the Id would be
/home/media/Music/ournas (which == /export/media)
As you can see the sample track is in the Samples folder under that folder.
On my Id I have the usual Music, Images and Video under that folder.

Logged

Shasta Mike

  • World Citizen
  • ***
  • Posts: 219
Re: Id File Structure and Library Migration
« Reply #2 on: June 06, 2022, 11:10:44 pm »

Quote from: bob
We don't encourage altering any of the paths setup.
There shouldn't be anything under the ~media/Music directory except ournas.
ournas is a symlink to the media partition where the all of the media on the Id is expected to be.
The actual path for that on disk is
/export/media
/export is the media partition and media is the writable folder under that.When you connect to the Id with samba/cifs you will see 2 shares.


Internal Storage which is the /export partition
and
External Storage which is where any devices mounted on the Id will appear (external drives, USB ..)

The default for MC on the Id is to import from /home/media/Music. You shouldn't add any more import paths to MC unless you add an external device to the Id.
  I understand now.  But, back at the ranch, I added the Music folder after Media to have a place to put all the music files not understanding that was a no-no. 

Quote from: bob
I am not an expert on the move/rename functions so I'm not sure how to answer that question. I'd be careful with that, you don't want two copies and you especially don't want things in the /home/media/Music folder that aren't under ournas or you will be filling the limited OS filesystem space.
  If the files are showing under ournas/Music should I delete the ones in Media/Music? 
Quote from: bob
As for cover art I store all mine in tags for files that support it and for ones that don't (like SACD) I store them in Folder.jpg so they travel with the media.
Good to know. 

Quote from: bob
You might want to try using the portable library function bringing over the windows library.
The prefix for the Id would be
/home/media/Music/ournas (which == /export/media)
As you can see the sample track is in the Samples folder under that folder.
On my Id I have the usual Music, Images and Video under that folder.
I didn't know what the Sample directory was and deleted it.  Another - oops! 

After flailing around on my own and now reading your very informative response, I have a much better idea of where I went wrong and what I should have done. 

Question: If I used option 77 from the Id Main Menu would that reset the Id to how I received it?  For example, would the file structures I have added be removed and the deleted folders be restored?  What exactly would Option 77 Revert to Default Settings do? 

Logged

bob

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 13452
Re: Id File Structure and Library Migration
« Reply #3 on: June 07, 2022, 12:15:58 pm »

  I understand now.  But, back at the ranch, I added the Music folder after Media to have a place to put all the music files not understanding that was a no-no. 
  If the files are showing under ournas/Music should I delete the ones in Media/Music?   Good to know. 
 I didn't know what the Sample directory was and deleted it.  Another - oops! 
When you say Media above, I'm not sure where you are referring to.
 
There is the users home directory media of which the path is /home/media which MC runs under.
Then the ournas symlink in /home/media/Music/ which points to /export/media directory.
You can put anything you want in ournas (aka /export/media, aka \\id_ip_address\InternalStorage\media), files, directories, whatever until you fill that drive partition.
You can check on how full it is by looking at the system monitor in the Id GUI, right-click desktop menu, then Start System Monitor OR look on the samba share with Properties on windows or GetInfo on Mac.
Quote

After flailing around on my own and now reading your very informative response, I have a much better idea of where I went wrong and what I should have done. 

Question: If I used option 77 from the Id Main Menu would that reset the Id to how I received it?  For example, would the file structures I have added be removed and the deleted folders be restored?  What exactly would Option 77 Revert to Default Settings do?
Option 77 should reset everything in the media user home directory to default. It will wipe anything under the Music directory there as well as reset MC's library to default and you'll need to re-enter the license.
Logged

Shasta Mike

  • World Citizen
  • ***
  • Posts: 219
Re: Id File Structure and Library Migration
« Reply #4 on: June 07, 2022, 01:26:34 pm »

Quote from: bob
When you say Media above, I'm not sure where you are referring to.
  I'm talking about what I see in Windows FE as in the attached picture.  Internal Storage/Media

Thanks to your explanations I am starting to get some idea of the file system on the Id.  It still is not 100% clear because I am a more visual and kinesthetic learner (I learn better by watching someone else do something and then do it myself) than by reading.

So, because the way the file folders are organized in the Id are not the way they display in Windows makes it hard for me to understand.  And, the way they display in the Id does not seem to be the way they are actually stored in the Id (ref your use of the term "symlink" which is new for me) is confusing.  I need to read what you wrote about 10 more times to hopefully get it.  Thanks!   

BTW - is there a visual cue when browsing folders in the Id to know what is a symlink folder and a folder that actually is storing data files?  And, is there someway to see all these folders in Windows?  So much is left out in the Windows display. 
Logged

bob

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 13452
Re: Id File Structure and Library Migration
« Reply #5 on: June 07, 2022, 03:46:53 pm »

  I'm talking about what I see in Windows FE as in the attached picture.  Internal Storage/Media
Ah, that's why I didn't get it. Everything is case sensitive on linux. So it's InternalStorage/media
Note that windows doesn't care when writing into the location about case but when MC on the Id will access them case will matter.
Quote
Thanks to your explanations I am starting to get some idea of the file system on the Id.  It still is not 100% clear because I am a more visual and kinesthetic learner (I learn better by watching someone else do something and then do it myself) than by reading.

So, because the way the file folders are organized in the Id are not the way they display in Windows makes it hard for me to understand.  And, the way they display in the Id does not seem to be the way they are actually stored in the Id (ref your use of the term "symlink" which is new for me) is confusing.  I need to read what you wrote about 10 more times to hopefully get it.  Thanks!   

BTW - is there a visual cue when browsing folders in the Id to know what is a symlink folder and a folder that actually is storing data files?  And, is there someway to see all these folders in Windows?  So much is left out in the Windows display.
As looking from windows, the ONLY filesystems you can see on the Id are
/export == InternalStorage
and
/media == ExternalStorage

You don't need to care about symlinks normally because you can't create or access them EXCEPT for when you manually use something like Move, Rename and Copy.

Anyway the only thing you need to care about when in MC on the Id is to put everything under ournas somewhere (or the place it symlinks to /export/media)
It can be all freeform there but what I do is have the standard Music, Images and Video under there.



Logged

Shasta Mike

  • World Citizen
  • ***
  • Posts: 219
Re: Id File Structure and Library Migration
« Reply #6 on: June 09, 2022, 10:45:41 am »

Quote from: bob
Anyway the only thing you need to care about when in MC on the Id is to put everything under ournas somewhere (or the place it symlinks to /export/media)
It can be all freeform there but what I do is have the standard Music, Images and Video under there.
On the last three albums from the PC using FE I put them in the /internal Storage/Media/ folder. On the Id I can see them in /export/media/ After running import I could add to the playlist and play them.  But, no matter what I do, the cover art will not display.  I have tried every combination of settings I can think of in Tools-Options (see attached pic for current) for folder settings and Store in Tags, etc.  Unless I use Rename to move the file over manually to /home/media/Music/ournas/Music the cover art will not display. 

Is there a simpler way to bring the music into the Id and get everything to Import automatically?  Doing all the Renaming and other searching for folders, etc takes forever.  I am still missing something fundamental in using MC on the Id and getting new music folders 1) onto the Id 2) and then into MC so I can: 3) add to Playlists and do other tagging operations (genre, etc). and see Cover Art.   

Hopefully, you/Bob or someone else will see my foolish mistake.  I have spent another 4-5 hours on three folders of music experimenting with different procedures but have gotten nowhere and now it is feeling like doing the same thing over and over and expecting different results. 
Logged

bob

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 13452
Re: Id File Structure and Library Migration
« Reply #7 on: June 09, 2022, 04:20:57 pm »

I've never messed with the rename, move, copy dialog.
People do that between platforms but it's more than I want to mess with.

I started with a clean library, copied my music to the Id under the \\InternalStorage\media share
All of my albums have either the cover art in the tags or in a Folder.jpg in the Album's folder.

Then I import that all on the Id.
When I did that I recreated the playlists.
There is probably a way to export them on windows and import them on the Id but I didn't try.
In settings, File Location->Cover Art->In same folder as the file (Folder.jpg on the Id that's case sensitive.)
For the Cover art I'm pretty sure it will pickup existing Folder.jpg that's just for new ones.

If you run into issues with file permissions under the Id's Custom System Configuration menu item 17 there is an option to fix ownership/permissions on the InternalStorage\media folder (and everything under it).





Logged

Shasta Mike

  • World Citizen
  • ***
  • Posts: 219
Re: Id File Structure and Library Migration
« Reply #8 on: June 09, 2022, 06:37:48 pm »


In settings, File Location->Cover Art->In same folder as the file (Folder.jpg on the Id that's case sensitive.)
For the Cover art I'm pretty sure it will pickup existing Folder.jpg that's just for new ones.

If you run into issues with file permissions under the Id's Custom System Configuration menu item 17 there is an option to fix ownership/permissions on the InternalStorage\media folder (and everything under it).
I tried the fix ownership/permissions option and that didn't do anything for displaying the Cover Art.  So, then I tried moving the Cover Art folder in options from /export/media/ to /export/media/Music where most files are located and I got a message "are you sure you want to do this?" and said yes and then MC locked up and I had to reboot the Id so gave up on this.  I have already checked "also store the  image in the files tag."  So far, I cannot get the Cover Art to display when importing to /export/media/ AKA Internal Storage/media/ in Win10 FE.  One of the more perplexing things is I highlight the songs in MC, use Cover Art > Add from file > find the file and JPG file > select the JPG and it still doesn't display. 

1. Am I dragging/dropping to the wrong place?
2. Do I have the file locations improperly set in Tool>Options? 
3. Is Import configured incorrectly?   

I just want to be able to drag and drop an album folder that has several flac files and includes the Cover Art in jpg format from my PC into the Id.  Then I want the Id to Auto-import it into MC somewhere where it is easy to go to it and then I went to send it to a playlist.  And, I want the Cover Art to display.   

Just spent another three hours trying various things that all fail.  Yikes! I am starting to think this will not work after spending probably 80 hours over the last month trying to set this Id up. But, actually I am pretty close and need to solve the issue of getting new folders to the Id, and importing them so they can be accessed and the cover art displays.   

Here's a data point.  On the folders that display Cover Art if I look in Tag info they are in /home/media/Music/ournas/Music and the folder that will not display cover art is in /export/media/   I don't understand at all what is going on behind the scenes where MC or the Id are moving files or pointing at different locations.  And, I don't understand how to set file locations to modify this behavior or set things up to take advantage of the behavior and not be in conflict with it.  I really wish I could understand what the Id is doing and what MC is doing when running import.  On my PC it seemed to all work seamless but on the Id it is very confusing to say the least. 

Appreciate the help - I really do because I am learning things.  I'm just not learning the things I need to learn to make this work. 
Logged

JimH

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 71212
  • where the buffalo roam
Re: Id File Structure and Library Migration
« Reply #9 on: June 10, 2022, 01:57:14 am »

Is your cover art embedded in the files? 
The wiki has a topic on cover art that explains your options.
Logged

Shasta Mike

  • World Citizen
  • ***
  • Posts: 219
Re: Id File Structure and Library Migration
« Reply #10 on: June 10, 2022, 10:39:25 am »

Is your cover art embedded in the files? 
The wiki has a topic on cover art that explains your options.
For each album there is a folder.  Inside the folder are flac files and one jpg with the cover art.  This is from the Wiki... "Alternatively you can link any image file by using the "Tools > Cover Art > Add From File..." command. In this case the filename does not matter - for example, Oxygene-1977.jpg will work." and confounding to me, this action will not bring up the cover art which is what I find perplexing.   What could block this action? 

The cover art is not consistently named.  In 90+% cases it is named cover.jpg   But, sometimes folder.jpg and in rare cases it might be something to do with the album.  I have several sources for music and they don't all use the same naming convention.   That will prevent some of the neater functions built into MC from working like the one associated with Folder.jpg
Logged

bob

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 13452
Re: Id File Structure and Library Migration
« Reply #11 on: June 10, 2022, 01:38:46 pm »

You really don't want your cover art folder to be in the place where it will get "imported".
That will likely produce a loop.
If you really want a separate cover art folder under /export/media
You should change the auto-import settings in MC and remove the default path that's there.
Make new ones like
/export/media/Music for music
/export/media/Images for pictures
and
/export/media/Video for video

The your cover art folder under /export/media/coverart (for example) won't get imported.

If you select "In same folder as the file"
The Folder: path should be immaterial.

So then you could do something like
Add cover art from file
then
Tree & View -> build missing thumbnails

I'm sure we can get this worked out.
If you have any more questions you don't have to wait until you've spent a bunch of time to ask.


Logged

Shasta Mike

  • World Citizen
  • ***
  • Posts: 219
Re: Id File Structure and Library Migration
« Reply #12 on: June 10, 2022, 04:12:36 pm »

Maybe if the current problem can be fixed the rest will follow suit.  Here are a couple pics that 1) show the file location of the folder and empty cover art.  3) I click on the cover art and get Tools.  I click that and choose Add from file.  You can see in the 2nd pic, I select cover art and then in that dialogue box click Open.  The dialogue box disappears and nothing happens.

Should I move the whole folder somewhere?  And, in the Id what is the best way to do that?  The only way I know is through MC using Rename, Move, ....

Or, is there something else to try?   
Logged

bob

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 13452
Re: Id File Structure and Library Migration
« Reply #13 on: June 10, 2022, 04:37:46 pm »

Assuming the cover art is OK (you checked it on windows)?
Perhaps there is an issue with the path on linux (which would be a MC bug) and not an actual problem with the cover art.
I see there are square brackets in the title. Any other non alpha characters? Can you post the whole path to that album?
Thanks.
Logged

Shasta Mike

  • World Citizen
  • ***
  • Posts: 219
Re: Id File Structure and Library Migration
« Reply #14 on: June 10, 2022, 05:06:25 pm »

Assuming the cover art is OK (you checked it on windows)?
Perhaps there is an issue with the path on linux (which would be a MC bug) and not an actual problem with the cover art.
I see there are square brackets in the title. Any other non alpha characters? Can you post the whole path to that album?
Thanks.
- Checked cover art in Windows and it displays fine. 
- here is the complete file name....  Stray Fossa [2022] Closer Than We'll Ever Know [Born Losers Records, WEB] [FLAC]

BTW - I can listen to the music just fine.  :)
Logged

bob

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 13452
Re: Id File Structure and Library Migration
« Reply #15 on: June 10, 2022, 05:22:54 pm »

The cover art may not use the same path checks.
I'll check on my end.
Pretty packed schedule though, I might not get it until Monday.
Logged

bob

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 13452
Re: Id File Structure and Library Migration
« Reply #16 on: June 13, 2022, 05:11:14 pm »

I tested this with exactly the same path as yours.

First I tried with a .m4a because that's what's in my library.
Along with 6. Liminal.m4a and with a cover.jpg in that same folder.
Removed my existing cover art (it was in my file tags too).
My cover art is set to be in the same directory as the music track as Folder.jpg.
I added the cover art from file cover.jpg
MC built a thumbnail right away and created a Folder.jpg alongside the cover.jpg
So that worked.
I thought maybe there is some issue with flac vs m4a so I put a flac in there and removed the m4a and Folder.jpg and imported again.
Removed existing cover art as before.
Imported the cover.jpg again.
MC didn't build the thumbnail right away, I don't know why. It was embedded in the file tags and I could see it there. Did a refresh of the view, still no thumb.
Exited and restarted MC and the thumb showed up.

Just duplicated that behavior with the "save also to the file's tag" disabled and it did the same thing.
You might try just clicking the exit button on the Id MC (It automatically restarts at the top of the next minute) and see if your thumbnails appear.

I also tried a little experiment with moving the cover art directory to the /export/media (InternalStorage/media share)
I removed my existing ournas import location from the Id's MC auto-import location.
I have directories called Music, Images and Video under /export/media (InternalStorage/media share)
Added those to the auto-import settings as /export/media/Music /export/media/Images and /export/media/Video.
All of my material is in those subdirs nothing should be imported now from /export/media
So then I moved my cover art directory from it's default location to /export/media/CoverArt
Everything went fine (I have about 20k files and it was fast).
So that just shows that you can store the cover art on the share and not in the album folders if you prefer.
Logged

Shasta Mike

  • World Citizen
  • ***
  • Posts: 219
Re: Id File Structure and Library Migration
« Reply #17 on: June 14, 2022, 12:59:55 am »

You might try just clicking the exit button on the Id MC (It automatically restarts at the top of the next minute) and see if your thumbnails appear.
  Tried this without success.  Thanks for running these different tests.
Logged

bob

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 13452
Re: Id File Structure and Library Migration
« Reply #18 on: June 14, 2022, 09:15:58 am »

  Tried this without success.  Thanks for running these different tests.
Sent you a PM.
Logged
Pages: [1]   Go Up