INTERACT FORUM

Please login or register.

Login with username, password and session length
Advanced search  
Pages: [1]   Go Down

Author Topic: Music Chart Data  (Read 12962 times)

rick.ca

  • Citizen of the Universe
  • *****
  • Posts: 3729
Music Chart Data
« on: December 11, 2010, 06:45:11 pm »

Thanks to mark_h and this topic for inspiring me to do this. For some time I've liked the idea of including chart data in my library, but wasn't sure how to go about it. Weekly chart data (what Mark is using) would be more than I would ever be interested in, but annual top 100 charts were of interest. I understood how to handle the data in MC, but wasn't sure how to import it (specifically, matching the data to files where artist and track names vary). And even if I could figure this out, I didn't like the idea of having to do for everything subsequently added to the library. So this is what I did...

  • Copied the last 50 years' Billboard Hot 100 charts into Excel.

  • Parsed the data to create the DOS commands: echo [data] > [year]\[rank] [artist] - [name].log.

  • Saved the commands as a batch file and ran it to create a dummy file for each chart entry.

  • Imported these files to MC, changed Media type from Data to Audio, and set Genre to charts.

  • Tagged Year, Track # (rank), Artist and (track) Name using Fill properties from filename.

  • Compared these tags to values actually used in the library and revised as necessary (mainly dealing with varying uses of "the" and "&", moving "featuring" data out of Name, etc.).

  • Set the custom nested list field Charts to Hot 100\(decade)\[Year]\[Track #] [Artist]: [Name]

  • I also have the Rolling Stone 500 collection, so I tagged those in the same manner.

  • In a list view sorted by Name and grouped by Artist, copied Charts from dummy files to the corresponding media files (also cross-tagging chart files for tracks that made two charts).

In a Panes View this results in a Charts pane that shows the entire chart for the selected year (i.e., all songs on the chart, whether or not in the library). By not selecting the "charts" Genre, only the media files in the library are shown in the view. These can be sorted in Charts order with duplicates removed to display a chart playlist.

It's essential, of course, the dummy files be excluded from any selection that's going to be played. For the most part, tagging them with the "charts" Genre works for me, but only because I use only primary genre classifications (rock, classical, jazz, etc.—with the Style tagged separately), and would vary rarely mix them. Even so, I expect to be tripping over these for a while as I revise smartlists and whatnot to exclude them where they're not wanted.
Logged

mark_h

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 1854
Re: Music Chart Data
« Reply #1 on: December 12, 2010, 03:42:56 am »

I'll be following your footsteps once I finish my weekly charts stuff.

The ability to do this sort of thing is what makes MC so special and makes our music collections so much more interesting.

I was also wondering about how to add dummy tracks for those that I cannot collect... think you have a solution there I could use.
Logged

Vincent Kars

  • Citizen of the Universe
  • *****
  • Posts: 1154
Re: Music Chart Data
« Reply #2 on: December 13, 2010, 04:28:00 am »

What about making an empty.mp3 and use a copy in step 2 to solve the dummy files problem?
Logged

mark_h

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 1854
Re: Music Chart Data
« Reply #3 on: December 13, 2010, 04:47:16 am »

What about making an empty.mp3 and use a copy in step 2 to solve the dummy files problem?

Surely an empty FLAC! :D  *grumble*no stinking lossy  mp3 in my library*grumble*
Logged

rick.ca

  • Citizen of the Universe
  • *****
  • Posts: 3729
Re: Music Chart Data
« Reply #4 on: December 13, 2010, 05:24:09 am »

What about making an empty.mp3 and use a copy in step 2 to solve the dummy files problem?

What's "the dummy files problem"?
Logged

Vincent Kars

  • Citizen of the Universe
  • *****
  • Posts: 1154
Re: Music Chart Data
« Reply #5 on: December 13, 2010, 07:40:05 am »

What I understand from the posting was that either you have the actual track or you don't.
If you don't you can filter it out or substitute a empty file.
Logged

Vincent Kars

  • Citizen of the Universe
  • *****
  • Posts: 1154
Re: Music Chart Data
« Reply #6 on: December 13, 2010, 09:48:46 am »

The dummy file works except that MC doesn't like empty FLAC's.
It hangs
Logged

rick.ca

  • Citizen of the Universe
  • *****
  • Posts: 3729
Re: Music Chart Data
« Reply #7 on: December 13, 2010, 07:05:09 pm »

Quote
What I understand from the posting was that either you have the actual track or you don't.

Sorry, I still don't understand what you're getting at. To be clear, I've completed this project. It works fine. There are no problems...

Quote
The dummy file works except that MC doesn't like empty FLAC's.
It hangs

...perhaps because I use the .LOG type to designate dummy files. And, as I stated, I take care to ensure MC is not asked to play them. If it is, it won't hang, but will open the file in a text editor.
Logged

tunetyme

  • Galactic Citizen
  • ****
  • Posts: 410
  • Have tunes will travel
Re: Music Chart Data
« Reply #8 on: December 14, 2010, 08:58:09 am »

One of the great problems with chart data is there are so many different charts.  Especially when you go back to the 50's and 60's.  There is a chart to promote every Genre.  I bought Joel Whitburn's "Top 40 hits" and he uses a couple of methods (sales, radio play and various charts) to determine Peak date and position.  This works great for "singles" but misses the album tracks. 
I also use Dave McLeer's "Hit Singles" that lists the top 20 songs for each month in the US and the UK.  The problem is tracking those songs that have been in the top 20 for multiple months.  As a result I haven't entered any chart data but I use ratings to help identify the songs that charted.  5 stars for the top 20, 4 stars for the top 40 and 3 stars for good or listenable album tracks, 2 stars if the really suck and 1 star if it is a duplicate song.

What I would like to be able to do is identify the songs/albums that I want to buy to fill out my collection.  Right now I use a Word doc that I take with me when I go shopping for music.  It is great for Albums but it is of limited use for individual songs.

In doing the weekly charts how are you resolving storing the info of a song that has charted for several weeks?

Is there a means to store all the songs/albums in MC that I don't have and I want to acquire? 

Tunetyme
Logged

mark_h

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 1854
Re: Music Chart Data
« Reply #9 on: December 14, 2010, 09:00:37 am »

For tracking weekly positions you'll want to check my thread which is referenced at the top of this thread.

Cheers,

Mark
Logged

Vincent Kars

  • Citizen of the Universe
  • *****
  • Posts: 1154
Re: Music Chart Data
« Reply #10 on: December 14, 2010, 09:18:55 am »

Is there a means to store all the songs/albums in MC that I don't have and I want to acquire? 

As you can see in reply #6, I applied the method as described in the opening post by rick.ca
This populates MC with all Billboard top 100 from 1946 on and all are dummy files
Replace the dummies with the files you do have.
Filer on e.g. duration=0 to get a list of files you don't have
Logged

Vincent Kars

  • Citizen of the Universe
  • *****
  • Posts: 1154
Re: Music Chart Data
« Reply #11 on: December 14, 2010, 09:24:52 am »

There are no problems...

You're method works.
However you wrote
Quote
It's essential, of course, the dummy files be excluded from any selection that's going to be played.
This is because they are non audio files.
My suggestion was to use empty audio files instead.
- you have the entire Top 100 in view
- press play
- all will be played and the empty ones will play very shortly
- no selection or exclusion needed

So no 'problem' just another work around for the songs you don't have
Am I clear enough by now?
Logged

Gl3nn

  • Galactic Citizen
  • ****
  • Posts: 384
Re: Music Chart Data
« Reply #12 on: December 14, 2010, 10:22:22 am »

One of the great problems with chart data is there are so many different charts.  Especially when you go back to the 50's and 60's.  There is a chart to promote every Genre.  I bought Joel Whitburn's "Top 40 hits" and he uses a couple of methods (sales, radio play and various charts) to determine Peak date and position.  This works great for "singles" but misses the album tracks. 
I also use Dave McLeer's "Hit Singles" that lists the top 20 songs for each month in the US and the UK.  The problem is tracking those songs that have been in the top 20 for multiple months.  As a result I haven't entered any chart data but I use ratings to help identify the songs that charted.  5 stars for the top 20, 4 stars for the top 40 and 3 stars for good or listenable album tracks, 2 stars if the really suck and 1 star if it is a duplicate song.

I struggled with finding data for years until I stumbled upon something called "the Whitburn Project" spreadsheet in '07 or '08, and I've been using it since (you can Google that name and find the site).  The data goes back to 1890!  I haven't been back since (I'm mostly interested in 40s-50s-60s) so I don't know whether it's current or not.

I notice that the Billboard site now has archives of their weekly publication going back to 1940.  Not exactly user-friendly for importing data to MC though... I presume that's some kind of paid subscription... but if you're searching a specific bit of info, that's as good as it gets.

I've been importing to MC manually and will probably continue to do so unless an effective external import directly to the individual track (with control) is somehow provided (a plugin?).  With some of the music of the 40s and 50s and the various takes by various artists, I wouldn't feel comfortable with an unsupervised automatic tie-in.
Logged

rick.ca

  • Citizen of the Universe
  • *****
  • Posts: 3729
Re: Music Chart Data
« Reply #13 on: December 14, 2010, 03:47:48 pm »

In doing the weekly charts how are you resolving storing the info of a song that has charted for several weeks?

By using nested data in a list field, any number of lists with any number of periods can be saved for each track. See Step 7 of my original post.

Quote
Is there a means to store all the songs/albums in MC that I don't have and I want to acquire?

While using dummy files provides a means for getting the chart data into MC via filenames, their purpose is also to provide the information for the songs that don't actually exist in the collection. MC's database is based on the existing media files, and there's currently no other way to do this. I use the same method to keep a record of all the movies I've seen, but do no own (because I saw them in the theatre or rented them).

Replace the dummies with the files you do have.

...but not before copying the chart data to the media files. ;)

I prefer to leave the dummy files intact, keeping them all separate from real files by tagging them with [Genre]=[Charts]. This results in the entire chart appearing in the Charts pane, while the view shows only the files that exist. There's then no need to filter or have MC play dummy files.

Quote
So no 'problem' just another work around for the songs you don't have
Am I clear enough by now?

Yes, thanks. So it's not a solution to a problem, just a alternate method one might prefer. I prefer my method, as I really don't want to be confused by mixing dummy with real files in the view. I'm not losing anything by doing so because the entire chart is shown in the Charts pane.

I struggled with finding data for years until I stumbled upon something called "the Whitburn Project" spreadsheet...

Wow! Thanks! Another 37,000 dummy files for me to import! :D

Quote
I notice that the Billboard site now has archives of their weekly publication going back to 1940.  Not exactly user-friendly for importing data to MC though... I presume that's some kind of paid subscription... but if you're searching a specific bit of info, that's as good as it gets.

Check the link I provided in Step 1 of my original post. ;)

Quote
I've been importing to MC manually and will probably continue to do so unless an effective external import directly to the individual track (with control) is somehow provided (a plugin?)...

Yes, but it's difficult to imagine how that might work. That's why I like the dummy file technique. It seems like a lot of work, but differences in the track identification data between source and destination have to be reconciled somehow. Once the dummy files have been imported, all the features of MC can be brought to bear on that task.
Logged

broncodan

  • World Citizen
  • ***
  • Posts: 212
Re: Music Chart Data
« Reply #14 on: December 14, 2010, 04:41:53 pm »

Anyway to make this simple??  I would be interested in the Whitburn information but not sure about parsing, programming, etc...

Wonder if anyone corresponded with King Sparta (see third party forum) - he use to have a plugin for the early version of MC that did this and it worked great.  I have a lot of data in my files from that - but it isn't as clean as using one list as noted here.

I would definitly be interested in doing this for my library if it was put out in easy steps.  Thanks for your work on this.
Logged

Gl3nn

  • Galactic Citizen
  • ****
  • Posts: 384
Re: Music Chart Data
« Reply #15 on: December 15, 2010, 09:35:03 am »

By using nested data in a list field, any number of lists with any number of periods can be saved for each track. See Step 7 of my original post.

...

Yes, but it's difficult to imagine how that might work. That's why I like the dummy file technique. It seems like a lot of work, but differences in the track identification data between source and destination have to be reconciled somehow. Once the dummy files have been imported, all the features of MC can be brought to bear on that task.


Thanks for your post, BTW.  I been re-reading it and I think I understand most of it.  I want to give it a whirl.  Step 9, however eludes me... how do you "copy" the Charts field to the media file?
Logged

rick.ca

  • Citizen of the Universe
  • *****
  • Posts: 3729
Re: Music Chart Data
« Reply #16 on: December 15, 2010, 12:36:51 pm »

Quote
Step 9, however eludes me... how do you "copy" the Charts field to the media file?

It's easier to understand if you're not expecting it to be automatic... ;)

...but it's not as bad as it might seem. You probably have your preferred method for one-at-a-time tagging, but this is how I did it:

  • Display a list which includes both the dummy files and the target media files.
  • Group by Artist and sort by Name—so the dummies and targets are adjacent.
  • Use the search wizard to restrict files to Only duplicates of [Artist] and [Name].
  • Open the Charts field in the tagging window.
  • Select the files which are to have the same data in Charts (some will have two dummy files because the song made the chart for two years; also, you may have duplicates of the media file).
  • The Charts data will appear in the tagging window. Change the check box(es) from filled (some files) to checked (all files).

So it ends up being a mouse-intensive and somewhat mindless task of repeatedly selecting files and clicking-off check boxes. Don't worry—you won't have to do it more than 100 times per year of chart data. ;D

I found Step 6 to be the most time-consuming one. And it being done properly, of course, helped make this one reliable and mindless—I was already confident the files would be matched properly by Artist and Name.
Logged

Gl3nn

  • Galactic Citizen
  • ****
  • Posts: 384
Re: Music Chart Data
« Reply #17 on: December 16, 2010, 12:46:27 am »

Thanks, Rick.  I get it now.  I'm going to start with a subset of years and do a trial run first though!  :)
Logged
Pages: [1]   Go Up