INTERACT FORUM

Please login or register.

Login with username, password and session length
Advanced search  
Pages: [1]   Go Down

Author Topic: Best way to import large library?  (Read 3562 times)

e-head

  • World Citizen
  • ***
  • Posts: 121
Best way to import large library?
« on: June 04, 2008, 09:57:02 am »

I think I'm finally ready to let MC manage all my music. I have a pretty large library. Can I just set up the root folder for auto-import and let 'er rip? Are there any potential problems I might run into? For instance, I remember reading something about "audio analysis" going a little haywire if there are lots of files.

Alternatively, I could drag and drop stuff and do it a little bit at a time. My only problem with this approach is I haven't figured out how to get MC to do the audio-analysis when using drag and drop.
Logged

hit_ny

  • Citizen of the Universe
  • *****
  • Posts: 3310
  • nothing more to say...
Re: Best way to import large library?
« Reply #1 on: June 04, 2008, 10:16:46 am »

Pretty large, eh :)

simplest way is import first, analyse later.

You can do both simultanesously but it just takes longer that way.

Try with a few to get a feel for how long it might take.

Another factor is whether your collection is on external drives or internal ones.

Much faster with internal.

Did you use any other program to manage your collectoin before MC ?

Maybe there is a way to dump that data into a format MC can understand and then all you would do is import that data. This could be the fastest option of the lot, import wise, preparing the data may take longer depending on how good your xml skills are.
Logged

steveklein

  • Galactic Citizen
  • ****
  • Posts: 478
Re: Best way to import large library?
« Reply #2 on: June 04, 2008, 10:22:33 am »

is it really that much faster on internal drives? mine didn't take too long to import from an external drive (~55,000 songs)...

and it certainly performs very well now that it is all imported.
Logged

e-head

  • World Citizen
  • ***
  • Posts: 121
Re: Best way to import large library?
« Reply #3 on: June 04, 2008, 11:02:23 am »

Hey hit_ny,

I'm a stubborn traditionalist who thought the only manager I needed before was total commander. You know... right click, "enqueue in foobar". Then I broke down and got an iPod, and after much hesitation I finally gave in and started using iTunes. I admit, using a manager wasn't as awful as I had anticipated. Only, iTunes is a far cry from perfect.

I had hoped there might be a way to translate the iTunes Library over somehow. My idea was to use iTunes export xml playlist function.

However, when you export a playlist in iTunes it doesn't really export any useful information. I'd love to have MC pick up the "last played", "rating", and "number of plays" fields. iTunes exports none of this with the playlist. Even if it did, I'm not sure if MC would pick it up when importing.

Is there an xml transformation one can do on the iTunes Library xml file itself? I know a little bit of coding. I managed to write up a vb script to extract all my iTunes cover art to folder.jpg. I take it MC has an API as well?

Right now I'm planning on just exporting 5 playlists, one for each rating, then importing them into MC and rating all the tracks that way. I was going to just give up on the "number of plays" and "last played" info.

Thanks for any info/pointers.
Logged

hit_ny

  • Citizen of the Universe
  • *****
  • Posts: 3310
  • nothing more to say...
Re: Best way to import large library?
« Reply #4 on: June 04, 2008, 02:56:01 pm »

is it really that much faster on internal drives? mine didn't take too long to import from an external drive (~55,000 songs)...

and it certainly performs very well now that it is all imported.
Which drive do you use and what interface does it use USB < firewire < esata ?

Quote
However, when you export a playlist in iTunes it doesn't really export any useful information. I'd love to have MC pick up the "last played", "rating", and "number of plays" fields. iTunes exports none of this with the playlist. Even if it did, I'm not sure if MC would pick it up when importing.

Is there an xml transformation one can do on the iTunes Library xml file itself?
If it does not include all the tags in the iTunes library then i'm wondering what the utility will be.

The only way may be to write the tags to the files in itunes and see if they come over when imported in MC.

OTOH does itunes allow to export its library in some format..maybe that might be more inclusive.
Logged

steveklein

  • Galactic Citizen
  • ****
  • Posts: 478
Re: Best way to import large library?
« Reply #5 on: June 04, 2008, 03:06:43 pm »

i've got a Lacie 500GB external drive. USB 2.0. works great.
Logged

e-head

  • World Citizen
  • ***
  • Posts: 121
Re: Best way to import large library?
« Reply #6 on: June 09, 2008, 02:18:16 pm »

I came up with a kludge for importing iTunes fields into MC.
MC has the ability to save and import files and their database fields in .mpl files. Fortunately, you can also import just database field changes... if, e.g., you have already imported the files into MC via the normal method.

So, if you can get iTunes to output a playlist in this .mpl format containing the fields you want to import into MC, you're basically set. I wrote up a perl script to import "last played", "play count", and "rating", which are about the only things I'm interested in.

I'm sure there are better ways of doing this.
Logged

Alex B

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 10121
  • The Cosmic Bird
Re: Best way to import large library?
« Reply #7 on: June 09, 2008, 03:29:33 pm »

MC has an iTunes xml importer. Export the iTunes library or just a playlist in xml format and import it to MC using the File > Import playlist command.

MC imports the Play Count and Rating field values without problems.

Unfortunately MC cannot interpret the Last Played field correctly. The resulting field value in MC seems to always be 01.01.1970 02:33 (at least on my system). Perhaps JRiver could fix this.
An iTunes XML file seems to contain a "play date" value in two different formats:
<key>Play Date</key><integer>3295897151</integer>
<key>Play Date UTC</key><date>2008-06-09T19:59:11Z</date>
Logged
The Cosmic Bird - a triple merger of galaxies: http://eso.org/public/news/eso0755

e-head

  • World Citizen
  • ***
  • Posts: 121
Re: Best way to import large library?
« Reply #8 on: June 09, 2008, 04:18:12 pm »

MC has an iTunes xml importer. Export the iTunes library or just a playlist in xml format and import it to MC using the File > Import playlist command.

MC imports the Play Count and Rating field values without problems.

Unfortunately MC cannot interpret the Last Played field correctly. The resulting field value in MC seems to always be 01.01.1970 02:33 (at least on my system). Perhaps JRiver could fix this.
An iTunes XML file seems to contain a "play date" value in two different formats:
<key>Play Date</key><integer>3295897151</integer>
<key>Play Date UTC</key><date>2008-06-09T19:59:11Z</date>

Thanks for the info!

I didn't know MC would import the play count and rating. I probably wasn't paying attention.
;)

I had to do some fiddling to translate the iTunes date (as returned by the PlayedDate() function) to a MC date (which seems to be just a "unix" timestamp).

I have no idea what that first format is. It isn't a unix timestamp, as that would be closer to something around "1213033511" for the UTC date that is in the next field. The reason for the 1970 is probably because the date is "out of range" for the unix "epoch", which began on Jan. 1, 1970.

The second date is in a far friendlier format.

Alright... after messing on IRC....
This may sound ridiculous, but apparently there is an "Apple epoch", corresponding to Steve Job's birthdate.
I was told to subtract 2082823200 and interpret as a unix timestamp.

Indeed:

3295897151 - 2082823200 = 1213073951, which I think is right.
Logged

Alex B

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 10121
  • The Cosmic Bird
Re: Best way to import large library?
« Reply #9 on: June 09, 2008, 05:07:17 pm »

Nice detective work. It should be easy to subtract "Mr. Jobs" and use the UNIX date since it is MC's internal format for the Last Played field. Now we just need to get this problem on Matt's to do list. :)

I have explained MC's internal date formats e.g. here:
http://yabb.jriver.com/interact/index.php?topic=43983.msg301174#msg301174
This thread contains lots of related info:
http://yabb.jriver.com/interact/index.php?topic=40963.msg279648#msg279648
Logged
The Cosmic Bird - a triple merger of galaxies: http://eso.org/public/news/eso0755

SwellGuy

  • Guest
Re: Best way to import large library?
« Reply #10 on: June 10, 2008, 10:05:19 am »

Alex,

Which version of the Play Date is not working (Play Date, Play Date UTC, or neither)?

Rick
Logged

Alex B

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 10121
  • The Cosmic Bird
Re: Best way to import large library?
« Reply #11 on: June 10, 2008, 10:45:26 am »

Both versions are included in an iTunes xml file. Apparently MC tries to import the last played value because the UNIX epoch date appers in the Last Played field inside MC.

Here's an example of the xml code:

Quote
      <dict>
         <key>Track ID</key><integer>239</integer>
         <key>Name</key><string>a test name</string>
         <key>Artist</key><string>a test artist</string>
         <key>Album Artist</key><string>Various</string>
         <key>Composer</key><string>maybe you</string>
         <key>Album</key><string>whatever</string>
         <key>Grouping</key><string>yes (or no)</string>
         <key>Genre</key><string>Techno</string>
         <key>Kind</key><string>MPEG audio file</string>
         <key>Size</key><integer>413956</integer>
         <key>Total Time</key><integer>15516</integer>
         <key>Disc Number</key><integer>2</integer>
         <key>Disc Count</key><integer>2</integer>
         <key>Track Number</key><integer>2</integer>
         <key>Track Count</key><integer>2</integer>
         <key>Year</key><integer>2005</integer>
         <key>BPM</key><integer>114</integer>
         <key>Date Modified</key><date>2008-06-09T20:01:05Z</date>
         <key>Date Added</key><date>2008-06-09T19:58:01Z</date>
         <key>Bit Rate</key><integer>188</integer>
         <key>Sample Rate</key><integer>44100</integer>
         <key>Comments</key><string>no comments</string>
         <key>Play Count</key><integer>3</integer>
         <key>Play Date</key><integer>3295898211</integer>
            <key>Play Date UTC</key><date>2008-06-09T20:16:51Z</date>

         <key>Rating</key><integer>60</integer>
         <key>Normalization</key><integer>4005</integer>
         <key>Compilation</key><true/>
         <key>Artwork Count</key><integer>1</integer>
         <key>Persistent ID</key><string>55816985997220F4</string>
         <key>Track Type</key><string>File</string>
         <key>Location</key><string>file://localhost/C:/test/2.mp3</string>
         <key>File Folder Count</key><integer>3</integer>
         <key>Library Folder Count</key><integer>1</integer>
      </dict>
Logged
The Cosmic Bird - a triple merger of galaxies: http://eso.org/public/news/eso0755

SwellGuy

  • Guest
Re: Best way to import large library?
« Reply #12 on: June 10, 2008, 11:44:02 am »

It'll be fixed in the next build.
Logged

Alex B

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 10121
  • The Cosmic Bird
Re: Best way to import large library?
« Reply #13 on: June 10, 2008, 11:48:12 am »

It'll be fixed in the next build.

Thanks!


Alright... after messing on IRC....
This may sound ridiculous, but apparently there is an "Apple epoch", corresponding to Steve Job's birthdate.
I was told to subtract 2082823200 and interpret as a unix timestamp.

Indeed:

3295897151 - 2082823200 = 1213073951, which I think is right.

Actually, that would make Mr. Jobs quite bit older than he looks.

The Apple epoch date is January 1, 1904.

It's explained e.g. here: http://ask.metafilter.com/5202/
Logged
The Cosmic Bird - a triple merger of galaxies: http://eso.org/public/news/eso0755

e-head

  • World Citizen
  • ***
  • Posts: 121
Re: Best way to import large library?
« Reply #14 on: June 11, 2008, 09:10:49 am »

I noticed the dates displayed in the iPod details pane are not correct either. I have a bunch of files at the bottom with the bogus date/time 12/31/1969 7:00 PM (I think these are the files I just listened to on the device). I don't think this is effecting the syncing operation though... even though I use smart playlists which sort on the "last played" date to update the iPod. This might be because I still have a lot of music with no "last played" date at all though.

They also seem to get imported correctly. From the "Last Played" column of the "Play Stats" view scheme, everything seems to look fine. Nothing with a last played date of 1969, which was a good year after all.
Logged
Pages: [1]   Go Up