Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[kstuart]

[Can a MOD please move the messages from:

http://yabb.jriver.com/interact/index.php?topic=77950

that are not concerning Embedded CUE sheets, to this thread - thanks !

They start with:

http://yabb.jriver.com/interact/index.php?topic=77950.msg531408#msg531408

but there are some posts after that message that are on topic of Embedded CUE sheets.]

[Frobozz]

On the subject of live music recordings from places like eTree and archive.org.  I've got a bunch.  Listening to live recordings is good music and fun listening.  Unfortunately none of them are imported into JRiver.  Too much work.  I play them by dragging a folder from File Explorer into the JRiver Playing Now window.  Not a very Media Center way of doing things, but it works.

One of my goals is to import those files into Media Center and get them properly tagged and organized.  But I don't know when I will ever get around to it.  The other thread mentioned a Foobar component for tagging eTree downloads.  I didn't know about that.  I'll have to try it and see how it does.

Importing into JRiver, organizing, and tagging etree downloads is more time consuming than doing the same for CD rips.  Importing the eTree stuff is on hold while I figure out how best to do it.

I don't know how many other JRiver users are also getting stuff from eTree or archive.org and dealing with these same issues. 

[kstuart]

As the other thread mentions, there is a Forum member who was possibly going to make a utility that would both convert from SHN to FLAC and also tag the FLAC files from the .txt file.

That sounds worth waiting for, to see if it does happen.

[MrC]

The Forum member is waiting more sample data.

What I want is at least 1/2 dozen collection samples that includes:

   - directory listing of the collection's files including the directory name
   - the accompanying text file

It is taking me too long to find and download sample shn files.

Copy/paste is no good, because it ruins line endings (and this already tripped me up once).

[glynor]

I'm going to try... Had some home-oriented chaos intervene, or it would have happened already.

[kstuart]

It is taking me too long to find and download sample shn files.

I'm confused - why would sample shn files be needed ?

The process would seem to be:

* Convert shn file to wav file
* Convert wav file to flac file
* Tag flac file with data in text file

The first two are well established, and the first one is simply

for %%i in (*.shn) do shorten.exe -x "%%i" "%%~ni.wav"

So, the only missing part is a command line way to tag flac files from data in a text file, since the only current way is GUI-based.

[MrC]

In case this helps, collect them via (on your Mac or Cygwin command line):

$ ls -lR samples > samples/ls-lR.txt
samples:
total 0
drwx------+ 1 MrC None 0 Feb 13 22:30 ddbb2010-12-11
drwx------+ 1 MrC None 0 Feb 13 22:30 galactic2011-06-12
drwxr-xr-x+ 1 MrC None 0 Feb 18 18:04 gd1966-03-25.38
drwx------+ 1 MrC None 0 Feb 13 22:30 gd1966-07-03.39
drwx------+ 1 MrC None 0 Feb 13 22:30 gd1969-01-17

samples/ddbb2010-12-11:
total 4
-rwx------+ 1 MrC None 1817 Feb 13 17:44 ddbb2010-12-11.textfile.txt

samples/galactic2011-06-12:
total 4
-rwx------+ 1 MrC None 1207 Feb 13 17:44 galactic2011-06-12.txt

samples/gd1966-03-25.38:
total 12736
-rwx------+ 1 MrC None     2364 Feb 13 21:27 gd1966-03-25.38.txt
-rwx------+ 1 MrC None 13036677 Feb 13 17:44 gd66-03-25d1t01.shn

samples/gd1966-07-03.39:
total 8
-rwx------+ 1 MrC None 5764 Feb 13 21:40 gd1966-07-03.39.txt

samples/gd1969-01-17:
total 1
-rwx------+ 1 MrC None 520 Feb 13 17:44 gd1969-01-17.txt


$ find samples -type f -size -200k -print0 | tar -cv -f samples.tar  --null -T -
samples/ddbb2010-12-11/ddbb2010-12-11.textfile.txt
samples/galactic2011-06-12/galactic2011-06-12.txt
samples/gd1966-03-25.38/gd1966-03-25.38.txt
samples/gd1966-07-03.39/gd1966-07-03.39.txt
samples/gd1969-01-17/gd1969-01-17.txt

and send the samples.tar.

[MrC]

I'm confused - why would sample shn files be needed ?

The process would seem to be:

* Convert shn file to wav file
* Convert wav file to flac file
* Tag flac file with data in text file

The first two are well established, and the first one is simply

for %%i in (*.shn) do shorten.exe -x "%%i" "%%~ni.wav"

So, the only missing part is a command line way to tag flac files from data in a text file, since the only current way is GUI-based.

I don't think you're confused about what I need - but rather unclear about your own goals and priorities: Do you want your files converted?  Or do you want to internally understand all the details of what somebody else needs to create a process for you to convert your files?  So when somebody offers to do something for me for free, I'll respond with How Can I Help?, rather than Why Do You Need This to Help You Help Me?

I have NOT asked for sample shn files, but sample NAMES of shn files.

I'll explain it one more time.  Because the data is crap.  It is ambiguous, it is incomplete, it is non-standard.  Reliably parsing arbitrary crap, incomplete ambiguous data files is hard.  And while it is easy to make it work for any single file, making it work generally is hard.  And every time I have to change an algorithm because I didn't have sufficient data (and so made reasonable but erroneous assumptions) means more wasted time.  So the more actual data I have to draw patterns from, to use in parsing and to drive a script, will make the job easier and more robust, with fewer reworks.

[kstuart]

I have NOT asked for sample shn files, but sample NAMES of shn files.

We are just having communications difficulties here.

You said: "It is taking me too long to find and download sample shn files."

Since you did not say "sample shn filenames", it seemed that you were downloading shn files.

That is why I was responding to.

[kstuart]

MrC wrote: "I did find a script and set of tools that purports to accomplish the file conversion and tag writing."

Can you post a link ?

Perhaps it is not necessary to duplicate effort.

[MrC]

We are just having communications difficulties here.

Maybe.  But I have asked privately, and in the forum, for:

So, if you want some help on this, I'm not going to work hard to find the input data.  If you have some reasonable number of samples in the correct formats (file and directory names, content, metadata files), I'm happy to see what I can come up with.

[kstuart]

The last interchange a week ago was... you stated:

" Given the few sample files, I can imagine there are dozens of variations that can thwart processing. "

and I replied:

" In general, one is dependent on people following conventions, and often they either don't bother or make mistakes.  "

You may have thought that there was an implied " If you can send me more sample text files, I will try to cover all the variations found in them. "   but you never said that.

===

Also, this script idea came out of the earlier exchange, where I was explaining why CUE file support in MC18 was a good thing - rather than just saying that everyone should split their image files:

"A lot of MC18 has to be legacy support - just for example  SHN files... in order to convert those SHN files to FLAC properly, I would need to get out foobar, because the only way to tag them is with the extension to the foobar component Mass Tagger called "Live Show Tagger" which can read the etree text file format.  To convert all legacy files is, I think, a little more time intensive than it is worth in the long run."

You volunteered:

" This can be done in one shot via command line shntool.  Ask if you want help."

and

" Yes, I'm talking about a quick script to parse the text files that accompany your .shn files, and convert those into tagged flacs. "

and I mentioned:

" Note that this would be of interest to others besides me (even glynor has shn files from etree), so hopefully it can be made available publicly, but, of course, that would be up to you. "

In other words, the only request was for support for CUE files in MC18.   But, it was expressed that the project would be appreciated by a community of people who have SHN files from etree.

I'll bundle together some more .txt files.

[kstuart]

Maybe.  But I have asked privately, and in the forum, for:
"So, if you want some help on this, I'm not going to work hard to find the input data.  If you have some reasonable number of samples in the correct formats (file and directory names, content, metadata files), I'm happy to see what I can come up with."

Actually, you stated:
"Simply put, these folks are nuts, and clearly have not heard of torrents (they are still stuck in mailing list announcements of ftp sites that come and go - yuck).

So, if you want some help on this, I'm not going to work hard to find the input data.  If you have some reasonable number of samples in the correct formats (file and directory names, content, metadata files), I'm happy to see what I can come up with."

and I replied:
"The torrents (19,350 of them) are at:

http://bt.etree.org/
"

In other words, since you mentioned "there are no torrents" and "find the input data", it seemed apparently what you were asking for.

[JimH]

kstuart,
Do I detect a sarcastic tone?

MrC is another user like yourself and just trying to help.  It's reasonable for him to ask you to do a little work.

[glynor]

MrC is another user like yourself and just trying to help.  It's reasonable for him to ask you to do a little work.

Yep.

He also privately asked me for the same thing, which (like I said) I'm trying to pull together, but I have a family emergency I'm dealing with this week (on top of an already jam-packed work week).

[kstuart]

kstuart,
Do I detect a sarcastic tone?

Absolutely not.

That sort of inference is one reason that I went back and laid out exactly what was said - as opposed to what people might have thought was said.

It's hard to mix technical details with "reading between the lines", so I don't read between the lines in a technical discussion.

I do understand that is hard to "turn off" when reading.

Note - I cannot do this for the other half of the conversation.  Meaning that I cannot know what things that I said which were worded poorly, and did not communicate what I thought I was saying.  So, please assume that many things I said were also poorly worded as well.

[MrC]

I spent a little time on this last night and have made good progress.  I had to rewrite the script a bit.

I'm currently:

 - pulling (and cross-checking) dates from the file name and venue content.
 - pulling (possible) venue
 - pulling artist from the venue
 - pulling track numbers and names (and assigning track numbers when they don't exist)
 - pulling disc or set number (and converting roman numerals to integers)
 - pulling comments (everything that remains un-parsed goes to comments)

Of the 10 sample info files, I'm parsing most of them correctly.   Until I get a feel for how many malformed files you have in actuality, I can't know if the file formatting is rare or common.  So I'll need your feedback.  You might just have to edit the few that are odd, and do something like insert some line breaks, or ...?

To Do:
  - fix a few edge cases
  - handle multi-disc sets
  - correlate parsed info with file names in a directory (and probably use file names to help parsing)
  - run the conversion tool and output the XML.

Code: [Select]

$ ls
Garcia730504/       galactic2011-06-12/ gd1966-07-03.39/    gd66-11-19/         jg1962-06-11/
ddbb2010-12-11/     gd1966-03-25.38/    gd1969-01-17/       ja-1970-05-07/      pfunk1976-09-26/

Code: [Select]

$ perl ../parse-etree.pl ddbb2010-12-11/ddbb2010-12-11.textfile.txt
**** ddbb2010-12-11/ddbb2010-12-11.textfile.txt
Artist: Dirty Dozen Brass Band
Date: 2010/12/11

____Venue____
Warren Haynes Christmas Jam
Asheville Civic Center
Asheville, North Carolina
------
Disc: 1
Track: 1    Name: Introduction By Warren Haynes
Track: 2    Name: Ain't Nothin' But A Party
Track: 3    Name: Papa Was A Rolling Stone >
Track: 4    Name: *#Spanish Moon
Track: 5    Name: *@$^Superstitious
Track: 6    Name: @$^+Use Me

____Comments____
Source: Schoeps CCM4V'S(din)>Lunatec V2>Benchmark AD2K>
Sound Devices 722 (24/48)
FOB/DFC/KFC/ZFC/AARP 40' From Stage, 6 1/2' High
DSP: Sound Devices 722>Sound Forge 9.0>CD Wave>flac(16)
Recorded By: Z-Man
Seeded By: Z-Man
 

* With Warren Haynes On Guitar And Vocals
# With Artemis Pyle On Percussion
@ With John Bell On Vocals
$ With Jojo On Keyboards
^ With Ron Holloway On Saxaphone
+ With Mike Barnes On Guitar

Flac Fingerprint

ddbb2010-12-11d01t01.flac:cff5e54a31ef32430a81e3144b722780
ddbb2010-12-11d01t02.flac:8b89092e90bb9545b4c132b49f4ea66d
ddbb2010-12-11d01t03.flac:bd9f90718a8ed579cfe775650699a169
ddbb2010-12-11d01t04.flac:479a48e7334dddc05864f333aa4781c7
ddbb2010-12-11d01t05.flac:fe76cb3c0cdec58f501f1d07bf07276c
ddbb2010-12-11d01t06.flac:d7f583a9e944076835288e2884b71021

Shntool
 
    length     expanded size    cdr  WAVE problems  fmt   ratio  filename
     1:21.01       14290796 B   ---   --   ---xx   flac  0.4037  ddbb2010-12-11d01t01.flac
    10:11.12      107808668 B   ---   --   ---xx   flac  0.5387  ddbb2010-12-11d01t02.flac
     7:35.40       80356124 B   ---   --   ---xx   flac  0.5394  ddbb2010-12-11d01t03.flac
     6:56.27       73445948 B   ---   --   ---xx   flac  0.5289  ddbb2010-12-11d01t04.flac
    10:12.53      108081500 B   ---   --   ---xx   flac  0.5402  ddbb2010-12-11d01t05.flac
     9:32.36      100985516 B   ---   --   ---xx   flac  0.5201  ddbb2010-12-11d01t06.flac
    45:49.19      484968552 B                            0.5298  (6 files)

Code: [Select]

$ perl ../parse-etree.pl galactic2011-06-12/galactic2011-06-12.txt
**** galactic2011-06-12/galactic2011-06-12.txt
Artist: Galactic
Date: 2011/06/12

____Venue____
What Stage
Bonnaroo Music Festival
Manchester, TN
------
Track: 1    Name: Intro
Track: 2    Name: Who Took The Happiness
Track: 3    Name: I Don't Know But It Sure Is Funky (Corey)
Track: 4    Name: Never Called You Crazy (Corey)
Track: 5    Name: Keep Steppin
Track: 6    Name: Boe Money
Track: 7    Name: You Don't Know (corey)
Track: 8    Name: Night People (corey)
Track: 9    Name: Cineramascope
Track: 10    Name: Balkan Wedding
Track: 11    Name: Total destruction to your mind (corey)
Track: 12    Name: Corey Plead for Gulf Coast
Track: 13    Name: Heart of Steel (corey)
Track: 14    Name: Baker's dozen >
Track: 15    Name: Stanley Drums >
Track: 16    Name: Baker's Dozen
Track: 17    Name: How Many More Times (Corey)

____Comments____
Source: Soundfield SPS200 > Sound Devices 788t (24/48)
Location: FOB, DFC, 8.0 ft high

DSP: Sound Devices 788t > Wave Agent > Neundo 3 (sps200 szone vst, mixdown) > Wavelab 6 (fades, tracks, dither w/UV22 HR, resample) > FLAC v1.2.1 (L8) as build in TLH v2.6.0.(Build 168)

Recorded by: Daniel Kopp
Taped & Transferred by Daniel Kopp (taperhead@gmail.com)
Setlist help by featbase.net

*A Team Dirty South Recording*

//Please support artist by purchasing their albums, merch, and seeing their live shows. Also, tell your friends about them.//

One Set:

Galactic was joined entire performance by Corey Glover (Living Colour) and Corey Henry (Rebirth Brass Band)


Code: [Select]

$ perl ../parse-etree.pl gd1966-03-25.38/gd*txt
**** gd1966-03-25.38/gd1966-03-25.38.txt
Artist: Grateful Dead
Date: 1966/03/25

____Venue____
Troupers Hall
Los Angeles, CA
------
Track: 1    Name: Stealin'
Track: 2    Name: Jam
Track: 3    Name: Hey Little One
Track: 4    Name: Hog For You  Baby
Track: 5    Name: You Don't Have To Ask
Track: 6    Name: Cold Rain And Snow
Track: 7    Name: Next Time You See Me

____Comments____


S:MR>DAT>ZA2>CD

Comments:
Thayer Jennings: After You Don't Have To Ask,
Garcia says: "Okay, we'll be back in just a  few scant minutes."
& Lesh: "We're gonna take a little break now.
Don't  forget: on the 6th, 7th, 8th, 9th..."(cut).
DeadBase X: After Hey Little One -- Bob: "Hey Little One".
After Cold Rain And Snow -- Bobby: "That was Cold Rain And Snow".

Transcribed by Teddy GoodBear: Troupers Hall was the meeting room
for a retired actor's club in  Hollywood. The rent for the
gig could'nt. have been much. We did  everything ourselves,
all in two days. We plastered handbills all over  Hollywood.
Stage decor was a few lengths of paisley cloth purchased that 
afternoon at a fabric store. For a box office, we had a card
table and a  cigar box. Our not-quite-full house must have had
over a hundred people; and when the  night was over, our net
take was $75. At 2 o'clock in the morning, we went  to Cantor's
Deli on Fairfax and spent it all on dinner for everybody --  with
desert. -- Grateful Dead Family Album (page 43).
http://www.pompano.net/~goodbear/concert_art/03_25_66-1.html 
http://www.pompano.net/~goodbear/concert_art/03_25_66-2.html 
To listen to the show using "Real Audio":
http://sugarmegs.org/ram/gd3-25-66.ram 
http://www.deadlegs.com/tunes/1960/1966/32566/32566.ram 

Recordings:
30 SB. The complete list circulates in SBD.
The beginning of Stealin' is clipped with a significant portion
missing. The end of Next Time You See Me cuts with a significant
portion missing. Sometimes this  recording incorrectly circulates
as 01/??/66 or 01/22/66. To see a recording box label,
(as per Michael Wanger letter, see  07/??/66), from Golden State
Recorders: 
http://www.pompano.net/~goodbear/concert_art/03_25_66-3.html 

Contributors:
Jeff Tiedrich <jeff@tiedrich.com>, Thayer Jennings<htj@geocities.com>,
Teddy GoodBear <GoodBear@GeoCities.com>,
David  Sorochty <scarletfire@erols.com>,
Michael Wanger <vidkid@well.com>, 
Grateful Dead Family Album--p.43, DeadLegs <www.deadlegs.com>,
SugarMegs:  Mike Maloney <www.SugarMegs.org>.


[kstuart]

Looks good !

So, is this at the point where we can do some alpha testing with it ?

[MrC]

A little more to do first.

I've now finished:

- pulling/using band abbreviations from etree (http://wiki.etree.org/index.php?page=BandAbbreviations)
  so that a missing artist can be added from the abbreviation in the file name, and used for cross-checking
  an artist listed in the venue.
- collecting file names from the directory, parsing them, and correlating them to tracks
- better track parsing

Now the tough file (missing artist, no line breaks to demarcate venue, non-track info in between track list, longer than usual venue):

Code: [Select]

$ cat jg1962-06-11.txt
Sleepy Hollow Hog Stompers
Boar's Head Coffee House
Jewish Community Center
San Carlos, CA
6/11/1962
Master Audience (?) Reel (Scotch 150) >
playback on Technics RS 1520 @ 7-1/2 ips
no dolby > Barry Glassberg's first gen
reel (Scotch 207 recorded via Otari MX5050
@ 7-1/2 ips no dolby) > playback on Technics
RS1506 > Tascam DA-40 DAT > Tascam CD-RW700
master CD > CDs > HP 9350i extraction using
EAC > .shn encoding using mkwACT.  Source
discs from Jay Jurina, extraction and .shn
encoding by Joe Jupille.  Sector boundary
alignment confirmed using shntool.
Single Disc (18) 62:52
--Set I--
1. Tuning [0:23]
2. Run Mountain [4:50]
3. Billy Grimes The Rover [3:51]
4. Cannonball Blues [4:06]
5. Devilish Mary [4:05]
6. Buck Dancer's Choice [2:16
7. Little Birdie [3:56]
8. Sally Goodin' [2:32]
9. Hold The Woodpile Down [4:08]
--Set II--
10. Crow Black Chicken [4:05]
11. The Johnson Boys [3:57]
12. Shady Grove [5:22]
13. Hop High Ladies [3:34]
14. Sweet Sunny South [4:28]
15. All Go Hungry Hash House [3:37]
16. Man Of Constant Sorrow (1) [2:25]
17. Rabbit Chase [3:00]
18. Three Men Went A-Hunting [2:10]
Notes:
(1) Garcia acapella
Times given are for tracks, not songs.
According to Deadlists, the band consists of
Jerry Garcia (guitar and banjo), Marshall Leicester
(banjo and guitar), and Dick Arnold (fiddle). The
performance took place at the 'new' Boar's Head
Coffeehouse, relocated in the San Carlos Jewish
Community Center.  Deadlists also hypothesizes
that this is an audience recording, possibly with
onstage mics.
is parsed as:

Code: [Select]

$  ../../parse-etree.pl jg1962-06-11*.txt            
Base: jg1962-06-11, band abbr: 'jg', Artist from lookup: 'Jerry Garcia'
Artist: Jerry Garcia
Date: 1962/06/11

____Venue____
Sleepy Hollow Hog Stompers
Boar's Head Coffee House
Jewish Community Center
San Carlos, CA

____Tracks____

 1 Tuning [0:23]                            jg1962-06-11t01.shn
 2 Run Mountain [4:50]                      jg1962-06-11t02.shn
 3 Billy Grimes The Rover [3:51]            jg1962-06-11t03.shn
 4 Cannonball Blues [4:06]                  jg1962-06-11t04.shn
 5 Devilish Mary [4:05]                     jg1962-06-11t05.shn
 6 Buck Dancer's Choice [2:16               jg1962-06-11t06.shn
 7 Little Birdie [3:56]                     jg1962-06-11t07.shn
 8 Sally Goodin' [2:32]                     jg1962-06-11t08.shn
 9 Hold The Woodpile Down [4:08]            jg1962-06-11t09.shn
10 Crow Black Chicken [4:05]                jg1962-06-11t10.shn
11 The Johnson Boys [3:57]                  jg1962-06-11t11.shn
12 Shady Grove [5:22]                       jg1962-06-11t12.shn
13 Hop High Ladies [3:34]                   jg1962-06-11t13.shn
14 Sweet Sunny South [4:28]                 jg1962-06-11t14.shn
15 All Go Hungry Hash House [3:37]          jg1962-06-11t15.shn
16 Man Of Constant Sorrow (1) [2:25]        jg1962-06-11t16.shn
17 Rabbit Chase [3:00]                      jg1962-06-11t17.shn
18 Three Men Went A-Hunting [2:10]          jg1962-06-11t18.shn

____Comments____
Master Audience (?) Reel (Scotch 150) >
playback on Technics RS 1520 @ 7-1/2 ips
no dolby > Barry Glassberg's first gen
reel (Scotch 207 recorded via Otari MX5050
@ 7-1/2 ips no dolby) > playback on Technics
RS1506 > Tascam DA-40 DAT > Tascam CD-RW700
master CD > CDs > HP 9350i extraction using
EAC > .shn encoding using mkwACT.  Source
discs from Jay Jurina, extraction and .shn
encoding by Joe Jupille.  Sector boundary
alignment confirmed using shntool.
Single Disc (18) 62:52
--Set I--
--Set II--
Notes:
(1) Garcia acapella
Times given are for tracks, not songs.
According to Deadlists, the band consists of
Jerry Garcia (guitar and banjo), Marshall Leicester
(banjo and guitar), and Dick Arnold (fiddle). The
performance took place at the 'new' Boar's Head
Coffeehouse, relocated in the San Carlos Jewish
Community Center.  Deadlists also hypothesizes
that this is an audience recording, possibly with
onstage mics.

[glynor]

Seriously awesome.

[Ekpen]

MrC.
Thanks.
If my memory serves me well, the million record set someone (maybe) Glynor posted a while ago, contains SHN files. Can this be used to extract or convert the contents to FLac?

Thanks again,

Ekpen.

[MrC]

MrC.
Thanks.
If my memory serves me well, the million record set someone (maybe) Glynor posted a while ago, contains SHN files. Can this be used to extract or convert the contents to FLac?

Dunno.  Can you find the reference?

[glynor]

MrC.
Thanks.
If my memory serves me well, the million record set someone (maybe) Glynor posted a while ago, contains SHN files. Can this be used to extract or convert the contents to FLac?

Thanks again,

Ekpen.

Not I.

[Ekpen]

 A million record set-. I am referring to the one posted on March 11- 2011. I was able to locate it last night.
It has been moved to MC 15 area.
I thought the record set has SHN files. I took a look at one of the massive archives then, I saw SHN extension.
Ekpen.

[kstuart]

A million record set-. I am referring to the one posted on March 11- 2011. I was able to locate it last night.
It has been moved to MC 15 area.
I thought the record set has SHN files. I took a look at one of the massive archives then, I saw SHN extension.
Ekpen.

On the basis of that information, I found the thread (indeed posted by glynor) at:

http://yabb.jriver.com/interact/index.php?topic=62892.0

[Ekpen]

Dunno.  Can you find the reference?

See my reply #23, if this may help.
Thanks.
Ekpen.

[MrC]

I saw it, but don't see any value in downloading 280GB of data.

[kstuart]

I just wanted to mention that it would be helpful if the parsing-text-file-and-writing-tags portion could be used separately from the converting-shn-files portion.

This is because there are a significant number of etree files that are already in flac format, but nevertheless have no tags, and have the information in the same text files (and the flac files are named with the same convention).

I remember the justification was something like "everyone has their own way of tagging files"... which, of course, doesn't consider that it would nevertheless be helpful to have the files originate with the information already in tags, and then users can still change it as they like...

[glynor]

On the basis of that information, I found the thread (indeed posted by glynor) at:

http://yabb.jriver.com/interact/index.php?topic=62892.0

Of course, I remember that now.  Didn't before.

[MrC]

I just wanted to mention that it would be helpful if the parsing-text-file-and-writing-tags portion could be used separately from the converting-shn-files portion.

Yup, already separate.

[Ekpen]

I saw it, but don't see any value in downloading 280GB of data.

You do not have to download all. If I remember correctly, there is a small set- sample within those sets "A" "B" etc, that may be of use to you.
I think those sets do contain SHN files, whether they are full blown  tracks or albums,  or not, I do not know.
Thanks.
Ekpen.

[MrC]

I now have working:

  - handle multi-disc sets
  - matching/stripping artist in Venue, using name found from etree

The file:

Code: [Select]

$ cat Garcia730504*txt
Jerry Garcia/Merl Saunders
5-4-1973
Homer's Warehouse, Palo Alto, CA
MSR>DAT3x>Delta DIO 2496>Soundforge>wav>flac
There are some imperfections that seem to be from the master reels.No adjustments made.


Disk 1 (Set 1) 
That's The Touch I Like
Expressway
Instrumental >
After Midnight >
It's No Use
That's All Right, Mama

Disk 2 (Set 2) 
It Takes A Lot To Laugh, It Takes A Train To Cry
Honey Chile
Lonely Avenue
Instrumental
Georgia On My Mind
The Night They Drove Old Dixie Down
How Sweet It Is

is parsed as:

Code: [Select]


$ perl   ../../parse-etree.pl  Garcia730504info.txt
____Artist____
Jerry Garcia/Merl Saunders

____Date____
1973/05/04

____Venue____
Homer's Warehouse, Palo Alto, CA
MSR>DAT3x>Delta DIO 2496>Soundforge>wav>flac
There are some imperfections that seem to be from the master reels.No adjustments made.

____Tracks____

Discs: 2

 1  1 That's The Touch I Like                  Garcia730504d1t1.shn
 1  2 Expressway                               Garcia730504d1t2.shn
 1  3 Instrumental >                           Garcia730504d1t3.shn
 1  4 After Midnight >                         Garcia730504d1t4.shn
 1  5 It's No Use                              Garcia730504d1t5.shn
 1  6 That's All Right, Mama                   Garcia730504d1t6.shn
 2  1 It Takes A Lot To Laugh, It Takes A Train To Cry
 2  2 Honey Chile                             
 2  3 Lonely Avenue                           
 2  4 Instrumental                             
 2  5 Georgia On My Mind                       
 2  6 The Night They Drove Old Dixie Down     
 2  7 How Sweet It Is

The venue in this sample file runs into the processing specs/comments.  I'll ignore that for now, and if there are more like this, I'll push such lines down into the comments.

[MrC]

What do you want for the Album value:

  - The directory name?
  - Some portion of the venue?

Are there any other tags you want filled in (such as Keyword: Live, or Genre, or Media Sub Type)?

[MrC]

Ok, I've now completed the .mpl creation, so tomorrow I'll be testing some importing.

[kstuart]

What do you want for the Album value:

  - The directory name?
  - Some portion of the venue?

Are there any other tags you want filled in (such as Keyword: Live, or Genre, or Media Sub Type)?

This is just my personal reaction:

For live albums, the Artist and Date is the important factor.  The city is the aspect that gives it some specificity and color.  A lot of cities have a "Fox Theater" or a "Palladium", so it doesn't tell you much.

So

Miles Davis
1967-11-06
Paris

are the three things that characterize it.

So, for me:

Album would be "Live in Paris November 6, 1967",
Artist would be "Miles Davis",
Date would be "1967-11-06",
Venue would have as much information as found in the text file.

I think that

Keyword - Live

couldn't hurt and could be useful for many, and also

Comment = Converted from SHN set NNNNN

where NNNNN is a several digit number that may be in some filenames (ask me if this is not in any example I sent you).

Sounds good !

[MrC]

Since I don't know the city with certainty, I can give you the venue portion for Album:

 Live in {venue} November 6, 1967

Since Venue is not a stock MC field, which field would you like for the captured venue?  Description?

Comments will contain everything not captured and stored elsewhere.

None of the files have the ##### number internally, but a couple of them in the file names I think:

   gd66-11-19.94106.txt
   gd1966-07-03.40.txt

Until I get a new version of MC, I'm at a standstill, since MC is crashing constantly with yesterday's build.

[kstuart]

I'm still on 129, because I am installing an OTA tuner and they don't work on any later version.

--

Yes, now that I think about it, there is no way for the parser to know what a "city" is.

--

The multi-digit number (called the "SHN ID") is a reference to a database at

http://db.etree.org/

Unfortunately, not every set of SHN files has them, otherwise you could just use the existing database fields from them.   Most GD sets have them, but shows of bands like "Dirty Dozen Brass Band" don't.

The number is helpful due to the presence of "upgrades".  For example, there are three different versions of 10-12-1968 and the most recent one is pitch-corrected.

--

For me personally, [Description] is where I put things that distinguish between different versions, so it would be good to have "converted from SHN files, ID # NNNNN" in that field.

And, it just occurred to me that [Places] would be good for the Venue information (trying to stay consistent with the field names, while using fields that are both searchable and saved).

[MrC]

Ok, good info.

The problem with Places and venue, is that venue will be multi-line, and a Large field works better, unless you want me to map multiple lines into a single line or a list of items in Places.  That's going to make it harder and uglier to edit.

[MrC]

We have some action!

From the following files and the generated foo.mpl:

Code: [Select]

$ ls
ddbb2010-12-11.textfile.txt  ddbb2010-12-11d01t02.flac    ddbb2010-12-11d01t04.flac    ddbb2010-12-11d01t06.flac
ddbb2010-12-11d01t01.flac    ddbb2010-12-11d01t03.flac    ddbb2010-12-11d01t05.flac    foo.mpl

$ cat *.txt
Dirty Dozen Brass Band
Warren Haynes Christmas Jam
Asheville Civic Center
Asheville, North Carolina
12/11/2010

Source: Schoeps CCM4V'S(din)>Lunatec V2>Benchmark AD2K>
Sound Devices 722 (24/48)
FOB/DFC/KFC/ZFC/AARP 40' From Stage, 6 1/2' High
DSP: Sound Devices 722>Sound Forge 9.0>CD Wave>flac(16)
Recorded By: Z-Man
Seeded By: Z-Man

Disc I

01 Introduction By Warren Haynes
02 Ain't Nothin' But A Party
03 Papa Was A Rolling Stone >
04 *#Spanish Moon
05 *@$^Superstitious
06 @$^+Use Me

* With Warren Haynes On Guitar And Vocals
# With Artemis Pyle On Percussion
@ With John Bell On Vocals
$ With Jojo On Keyboards
^ With Ron Holloway On Saxaphone
+ With Mike Barnes On Guitar

Flac Fingerprint

ddbb2010-12-11d01t01.flac:cff5e54a31ef32430a81e3144b722780
ddbb2010-12-11d01t02.flac:8b89092e90bb9545b4c132b49f4ea66d
ddbb2010-12-11d01t03.flac:bd9f90718a8ed579cfe775650699a169
ddbb2010-12-11d01t04.flac:479a48e7334dddc05864f333aa4781c7
ddbb2010-12-11d01t05.flac:fe76cb3c0cdec58f501f1d07bf07276c
ddbb2010-12-11d01t06.flac:d7f583a9e944076835288e2884b71021

Shntool
 
    length     expanded size    cdr  WAVE problems  fmt   ratio  filename
     1:21.01       14290796 B   ---   --   ---xx   flac  0.4037  ddbb2010-12-11d01t01.flac
    10:11.12      107808668 B   ---   --   ---xx   flac  0.5387  ddbb2010-12-11d01t02.flac
     7:35.40       80356124 B   ---   --   ---xx   flac  0.5394  ddbb2010-12-11d01t03.flac
     6:56.27       73445948 B   ---   --   ---xx   flac  0.5289  ddbb2010-12-11d01t04.flac
    10:12.53      108081500 B   ---   --   ---xx   flac  0.5402  ddbb2010-12-11d01t05.flac
     9:32.36      100985516 B   ---   --   ---xx   flac  0.5201  ddbb2010-12-11d01t06.flac
    45:49.19      484968552 B                            0.5298  (6 files)


[kstuart]

That looks awesome ! 

--

I guess then put the venue information in Description.  Put the aforementioned stuff for Description first, then start a new line for the venue information.

(Using a sortable field makes it more easy to find information, in case you wanted Shuffle Play to do "only Avalon Ballroom recordings".   Using a saved-to-tags field means that one can re-do an MC18 installation from scratch.  I spent a lot of time yesterday entering info into MC18 database for cue-and-single-flac-file albums -- and none of that data gets saved back.  So, I realized I should avoid info that is not saved as tags, as much as possible - I will probably split those files some time later).

[MrC]

Ok, time to finish this up...

The conversion and mpl file writing code is now done.  I've successfully converted shn files to flac and written the mpl.  An imported mpl will pull in the corresponding flac files from the directory.  (see attached).

A new option (-c) will cause shn --> flac conversion.  When conversion occurs, filename suffixes in the mpl file are changed from .shn to .flac.  Without conversion (i.e. only writing an mpl file), suffixes remain as .shn.  This way, you can pull data from the info files to update your .shn files, and convert and pull tags for the flac files too.

The tool is run on a single directory of files.  It is given a single info file as an argument.  Flac and mpl files will be created in the same directory as the info and shn files.  Shn or flac files are not mandatory, but their presence helps better info file parsing (due to their typical naming conventions).

How do you want me to handle overwriting existing flac files?  How about a -f option to force overwrite, otherwise abort the conversion for the file?

[MrC]

@glynor - Do you have a couple of sample info files and directory listings or md5 files for me ?  I'd like a few more test samples.

[glynor]

@glynor - Do you have a couple of sample info files and directory listings or md5 files for me ?  I'd like a few more test samples.

I'll dig through this weekend.

Sorry... I've been a stranger.  Work has been nutty.

[kstuart]

Hmm... If this tool could convert CUE files to MPL files, it could kill off the biggest existing perpetual bug in MediaCenter....

although I just saw this just now:

" Fixed: When importing a FLAC with embedded CUE file, the last track could have the wrong duration. "

[MrC]

Hmm... If this tool could convert CUE files to MPL files, it could kill off the biggest existing perpetual bug in MediaCenter....

Let's get the shn / info files done first...

[MrC]

Bump:

How do you want me to handle overwriting existing flac files?  How about a -f option to force overwrite, otherwise abort the conversion for the file?

[kstuart]

That sounds fine.   In general, it's always good to default to not writing, in case one triggers by accident.

[MrC]

Ok, time for you to test.  Here's what you need:

1. Download and install ActivePerl (or Cygwin tools - include perl and flac, and skip step 5)

2. Download and unzip the attached parse_etree.pl script and save it somewhere convenient.

3. shntool from http://www.etree.org/shnutils/shntool/
    http://www.etree.org/shnutils/shntool/dist/win32/shntool-3.0.10.zip

4. shorten from http://www.etree.org/shnutils/shorten/
    http://www.etree.org/shnutils/shorten/dist/win32/shorten-3.6.1.zip

5. flac from http://flac.sourceforge.net/
    http://sourceforge.net/projects/flac/files/flac-win/flac-1.2.1-win/flac-1.2.1-win.zip/download

6. Place the shntool, shorten, and flac executables (and dlls) in some folder that is in your PATH.  Or modify your PATH to include the directory where you store these files.

7. Open a command window and run the command.  The command syntax is:

   perl.exe parse_etree.pl [options] path\to\infofile

   Example:  perl.exe parse-etree.pl -cf  music\gd1966-07-03.39\gd1966-07-03.39.txt

The current options are:

      c=convert shn to flac (shn files are remain and are untouched)
      d=debug
      f=force overwrite of flac files
      m=don't write mpl
      v=verbose (output album info)

Without any options, only the MPL file will be written.  Use -c to cause conversion of .shn to .flac.  Use -f to force overwriting existing flac files.  If you want to see how the the info file is parsed, use just -v (so -vm will just output the parsing to the console and do nothing else).

The info file and .shn or .flac files must exist in the same directory.

8. In MC, use File > Import Playlist to import the newly created mpl file.  This will import the files referenced inside, and update the tags in MC.

If there are no shn files, but flac files already exist, you can still run parse_etree on an info file to create an mpl file, which can then be imported to add tags to existing flac files (which will also be imported).

If you have trouble, I'll need the info file, a directory listing of your files, and debug output wouldn't hurt.  Given the free-format of the info files, I'm sure there will be plenty of adjustments required.

Edit: attachment removed - see latest thread.

[kstuart]

The ActivePerl aspect is borderline, in that it is okay for individuals to use it for their own purposes.  Given their license wording, they would be okay with etree distributing it, but not okay with JRiver (non-commercial vs commercial).

So, it is good that you did not link to it explicitly.

[MrC]

There is nothing problematic with linking to the ActivePerl software download page or binary (in fact, I simply forgot to do so, as I have done so on other occasions).

I wrote a piece of code that might be of some use to you and other JRiver users.  Distribution of ancillary software, especially by other organizations, is entirely out of the realm and scope of this thread.