INTERACT FORUM

More => Old Versions => JRiver Media Center 18 for Windows => Topic started by: kstuart on February 14, 2013, 07:09:21 pm

Title: Converting SHN files to FLAC files
Post by: kstuart on February 14, 2013, 07:09:21 pm
[Can a MOD please move the messages from:

http://yabb.jriver.com/interact/index.php?topic=77950

that are not concerning Embedded CUE sheets, to this thread - thanks !

They start with:

http://yabb.jriver.com/interact/index.php?topic=77950.msg531408#msg531408

but there are some posts after that message that are on topic of Embedded CUE sheets.]
Title: Re: Converting SHN files to FLAC files
Post by: Frobozz on February 18, 2013, 06:15:34 pm
On the subject of live music recordings from places like eTree and archive.org.  I've got a bunch.  Listening to live recordings is good music and fun listening.  Unfortunately none of them are imported into JRiver.  Too much work.  I play them by dragging a folder from File Explorer into the JRiver Playing Now window.  Not a very Media Center way of doing things, but it works.

One of my goals is to import those files into Media Center and get them properly tagged and organized.  But I don't know when I will ever get around to it.  The other thread mentioned a Foobar component for tagging eTree downloads.  I didn't know about that.  I'll have to try it and see how it does.

Importing into JRiver, organizing, and tagging etree downloads is more time consuming than doing the same for CD rips.  Importing the eTree stuff is on hold while I figure out how best to do it.

I don't know how many other JRiver users are also getting stuff from eTree or archive.org and dealing with these same issues. 
Title: Re: Converting SHN files to FLAC files
Post by: kstuart on February 18, 2013, 08:47:09 pm
As the other thread mentions, there is a Forum member who was possibly going to make a utility that would both convert from SHN to FLAC and also tag the FLAC files from the .txt file.

That sounds worth waiting for, to see if it does happen.
Title: Re: Converting SHN files to FLAC files
Post by: MrC on February 19, 2013, 12:41:09 pm
The Forum member is waiting more sample data.

What I want is at least 1/2 dozen collection samples that includes:

   - directory listing of the collection's files including the directory name
   - the accompanying text file

It is taking me too long to find and download sample shn files.

Copy/paste is no good, because it ruins line endings (and this already tripped me up once).
Title: Re: Converting SHN files to FLAC files
Post by: glynor on February 19, 2013, 12:50:08 pm
I'm going to try... Had some home-oriented chaos intervene, or it would have happened already.
Title: Re: Converting SHN files to FLAC files
Post by: kstuart on February 19, 2013, 01:08:22 pm

It is taking me too long to find and download sample shn files.

I'm confused - why would sample shn files be needed ?

The process would seem to be:

* Convert shn file to wav file
* Convert wav file to flac file
* Tag flac file with data in text file

The first two are well established, and the first one is simply

for %%i in (*.shn) do shorten.exe -x "%%i" "%%~ni.wav"

So, the only missing part is a command line way to tag flac files from data in a text file, since the only current way is GUI-based.
Title: Re: Converting SHN files to FLAC files
Post by: MrC on February 19, 2013, 01:12:22 pm
In case this helps, collect them via (on your Mac or Cygwin command line):

$ ls -lR samples > samples/ls-lR.txt
samples:
total 0
drwx------+ 1 MrC None 0 Feb 13 22:30 ddbb2010-12-11
drwx------+ 1 MrC None 0 Feb 13 22:30 galactic2011-06-12
drwxr-xr-x+ 1 MrC None 0 Feb 18 18:04 gd1966-03-25.38
drwx------+ 1 MrC None 0 Feb 13 22:30 gd1966-07-03.39
drwx------+ 1 MrC None 0 Feb 13 22:30 gd1969-01-17

samples/ddbb2010-12-11:
total 4
-rwx------+ 1 MrC None 1817 Feb 13 17:44 ddbb2010-12-11.textfile.txt

samples/galactic2011-06-12:
total 4
-rwx------+ 1 MrC None 1207 Feb 13 17:44 galactic2011-06-12.txt

samples/gd1966-03-25.38:
total 12736
-rwx------+ 1 MrC None     2364 Feb 13 21:27 gd1966-03-25.38.txt
-rwx------+ 1 MrC None 13036677 Feb 13 17:44 gd66-03-25d1t01.shn

samples/gd1966-07-03.39:
total 8
-rwx------+ 1 MrC None 5764 Feb 13 21:40 gd1966-07-03.39.txt

samples/gd1969-01-17:
total 1
-rwx------+ 1 MrC None 520 Feb 13 17:44 gd1969-01-17.txt


$ find samples -type f -size -200k -print0 | tar -cv -f samples.tar  --null -T -
samples/ddbb2010-12-11/ddbb2010-12-11.textfile.txt
samples/galactic2011-06-12/galactic2011-06-12.txt
samples/gd1966-03-25.38/gd1966-03-25.38.txt
samples/gd1966-07-03.39/gd1966-07-03.39.txt
samples/gd1969-01-17/gd1969-01-17.txt

and send the samples.tar.
Title: Re: Converting SHN files to FLAC files
Post by: MrC on February 19, 2013, 01:27:13 pm
I'm confused - why would sample shn files be needed ?

The process would seem to be:

* Convert shn file to wav file
* Convert wav file to flac file
* Tag flac file with data in text file

The first two are well established, and the first one is simply

for %%i in (*.shn) do shorten.exe -x "%%i" "%%~ni.wav"

So, the only missing part is a command line way to tag flac files from data in a text file, since the only current way is GUI-based.

I don't think you're confused about what I need - but rather unclear about your own goals and priorities: Do you want your files converted?  Or do you want to internally understand all the details of what somebody else needs to create a process for you to convert your files?  So when somebody offers to do something for me for free, I'll respond with How Can I Help?, rather than Why Do You Need This to Help You Help Me?

I have NOT asked for sample shn files, but sample NAMES of shn files.

I'll explain it one more time.  Because the data is crap.  It is ambiguous, it is incomplete, it is non-standard.  Reliably parsing arbitrary crap, incomplete ambiguous data files is hard.  And while it is easy to make it work for any single file, making it work generally is hard.  And every time I have to change an algorithm because I didn't have sufficient data (and so made reasonable but erroneous assumptions) means more wasted time.  So the more actual data I have to draw patterns from, to use in parsing and to drive a script, will make the job easier and more robust, with fewer reworks.
Title: Re: Converting SHN files to FLAC files
Post by: kstuart on February 19, 2013, 02:19:31 pm

I have NOT asked for sample shn files, but sample NAMES of shn files.

We are just having communications difficulties here.

You said: "It is taking me too long to find and download sample shn files."

Since you did not say "sample shn filenames", it seemed that you were downloading shn files.

That is why I was responding to.
Title: Re: Converting SHN files to FLAC files
Post by: kstuart on February 19, 2013, 02:22:15 pm
MrC wrote: "I did find a script and set of tools that purports to accomplish the file conversion and tag writing."

Can you post a link ?

Perhaps it is not necessary to duplicate effort.
Title: Re: Converting SHN files to FLAC files
Post by: MrC on February 19, 2013, 02:23:24 pm
We are just having communications difficulties here.

Maybe.  But I have asked privately, and in the forum, for:

So, if you want some help on this, I'm not going to work hard to find the input data.  If you have some reasonable number of samples in the correct formats (file and directory names, content, metadata files), I'm happy to see what I can come up with.
Title: Re: Converting SHN files to FLAC files
Post by: kstuart on February 19, 2013, 03:14:04 pm
The last interchange a week ago was... you stated:

" Given the few sample files, I can imagine there are dozens of variations that can thwart processing. "

and I replied:

" In general, one is dependent on people following conventions, and often they either don't bother or make mistakes.  "

You may have thought that there was an implied " If you can send me more sample text files, I will try to cover all the variations found in them. "   but you never said that.

===

Also, this script idea came out of the earlier exchange, where I was explaining why CUE file support in MC18 was a good thing - rather than just saying that everyone should split their image files:

"A lot of MC18 has to be legacy support - just for example  SHN files... in order to convert those SHN files to FLAC properly, I would need to get out foobar, because the only way to tag them is with the extension to the foobar component Mass Tagger called "Live Show Tagger" which can read the etree text file format.  To convert all legacy files is, I think, a little more time intensive than it is worth in the long run."

You volunteered:

" This can be done in one shot via command line shntool.  Ask if you want help."

and

" Yes, I'm talking about a quick script to parse the text files that accompany your .shn files, and convert those into tagged flacs. "

and I mentioned:

" Note that this would be of interest to others besides me (even glynor has shn files from etree), so hopefully it can be made available publicly, but, of course, that would be up to you. "

In other words, the only request was for support for CUE files in MC18.   But, it was expressed that the project would be appreciated by a community of people who have SHN files from etree.

I'll bundle together some more .txt files.
Title: Re: Converting SHN files to FLAC files
Post by: kstuart on February 19, 2013, 03:20:37 pm
Maybe.  But I have asked privately, and in the forum, for:
"So, if you want some help on this, I'm not going to work hard to find the input data.  If you have some reasonable number of samples in the correct formats (file and directory names, content, metadata files), I'm happy to see what I can come up with."
Actually, you stated:
"Simply put, these folks are nuts, and clearly have not heard of torrents (they are still stuck in mailing list announcements of ftp sites that come and go - yuck).

So, if you want some help on this, I'm not going to work hard to find the input data.  If you have some reasonable number of samples in the correct formats (file and directory names, content, metadata files), I'm happy to see what I can come up with."

and I replied:
"The torrents (19,350 of them) are at:

http://bt.etree.org/
"

In other words, since you mentioned "there are no torrents" and "find the input data", it seemed apparently what you were asking for.
Title: Re: Converting SHN files to FLAC files
Post by: JimH on February 19, 2013, 03:31:44 pm
kstuart,
Do I detect a sarcastic tone?

MrC is another user like yourself and just trying to help.  It's reasonable for him to ask you to do a little work.
Title: Re: Converting SHN files to FLAC files
Post by: glynor on February 19, 2013, 03:35:15 pm
MrC is another user like yourself and just trying to help.  It's reasonable for him to ask you to do a little work.

Yep.

He also privately asked me for the same thing, which (like I said) I'm trying to pull together, but I have a family emergency I'm dealing with this week (on top of an already jam-packed work week).
Title: Re: Converting SHN files to FLAC files
Post by: kstuart on February 19, 2013, 04:45:04 pm
kstuart,
Do I detect a sarcastic tone?

Absolutely not.

That sort of inference is one reason that I went back and laid out exactly what was said - as opposed to what people might have thought was said.

It's hard to mix technical details with "reading between the lines", so I don't read between the lines in a technical discussion.

I do understand that is hard to "turn off" when reading.

Note - I cannot do this for the other half of the conversation.  Meaning that I cannot know what things that I said which were worded poorly, and did not communicate what I thought I was saying.  So, please assume that many things I said were also poorly worded as well.
Title: Re: Converting SHN files to FLAC files
Post by: MrC on March 05, 2013, 12:09:11 pm
I spent a little time on this last night and have made good progress.  I had to rewrite the script a bit.

I'm currently:

 - pulling (and cross-checking) dates from the file name and venue content.
 - pulling (possible) venue
 - pulling artist from the venue
 - pulling track numbers and names (and assigning track numbers when they don't exist)
 - pulling disc or set number (and converting roman numerals to integers)
 - pulling comments (everything that remains un-parsed goes to comments)

Of the 10 sample info files, I'm parsing most of them correctly.   Until I get a feel for how many malformed files you have in actuality, I can't know if the file formatting is rare or common.  So I'll need your feedback.  You might just have to edit the few that are odd, and do something like insert some line breaks, or ...?

To Do:
  - fix a few edge cases
  - handle multi-disc sets
  - correlate parsed info with file names in a directory (and probably use file names to help parsing)
  - run the conversion tool and output the XML.

Code: [Select]
$ ls
Garcia730504/       galactic2011-06-12/ gd1966-07-03.39/    gd66-11-19/         jg1962-06-11/
ddbb2010-12-11/     gd1966-03-25.38/    gd1969-01-17/       ja-1970-05-07/      pfunk1976-09-26/


Code: [Select]
$ perl ../parse-etree.pl ddbb2010-12-11/ddbb2010-12-11.textfile.txt
**** ddbb2010-12-11/ddbb2010-12-11.textfile.txt
Artist: Dirty Dozen Brass Band
Date: 2010/12/11

____Venue____
Warren Haynes Christmas Jam
Asheville Civic Center
Asheville, North Carolina
------
Disc: 1
Track: 1    Name: Introduction By Warren Haynes
Track: 2    Name: Ain't Nothin' But A Party
Track: 3    Name: Papa Was A Rolling Stone >
Track: 4    Name: *#Spanish Moon
Track: 5    Name: *@$^Superstitious
Track: 6    Name: @$^+Use Me

____Comments____
Source: Schoeps CCM4V'S(din)>Lunatec V2>Benchmark AD2K>
Sound Devices 722 (24/48)
FOB/DFC/KFC/ZFC/AARP 40' From Stage, 6 1/2' High
DSP: Sound Devices 722>Sound Forge 9.0>CD Wave>flac(16)
Recorded By: Z-Man
Seeded By: Z-Man
 

* With Warren Haynes On Guitar And Vocals
# With Artemis Pyle On Percussion
@ With John Bell On Vocals
$ With Jojo On Keyboards
^ With Ron Holloway On Saxaphone
+ With Mike Barnes On Guitar

Flac Fingerprint

ddbb2010-12-11d01t01.flac:cff5e54a31ef32430a81e3144b722780
ddbb2010-12-11d01t02.flac:8b89092e90bb9545b4c132b49f4ea66d
ddbb2010-12-11d01t03.flac:bd9f90718a8ed579cfe775650699a169
ddbb2010-12-11d01t04.flac:479a48e7334dddc05864f333aa4781c7
ddbb2010-12-11d01t05.flac:fe76cb3c0cdec58f501f1d07bf07276c
ddbb2010-12-11d01t06.flac:d7f583a9e944076835288e2884b71021

Shntool
 
    length     expanded size    cdr  WAVE problems  fmt   ratio  filename
     1:21.01       14290796 B   ---   --   ---xx   flac  0.4037  ddbb2010-12-11d01t01.flac
    10:11.12      107808668 B   ---   --   ---xx   flac  0.5387  ddbb2010-12-11d01t02.flac
     7:35.40       80356124 B   ---   --   ---xx   flac  0.5394  ddbb2010-12-11d01t03.flac
     6:56.27       73445948 B   ---   --   ---xx   flac  0.5289  ddbb2010-12-11d01t04.flac
    10:12.53      108081500 B   ---   --   ---xx   flac  0.5402  ddbb2010-12-11d01t05.flac
     9:32.36      100985516 B   ---   --   ---xx   flac  0.5201  ddbb2010-12-11d01t06.flac
    45:49.19      484968552 B                            0.5298  (6 files)

Code: [Select]
$ perl ../parse-etree.pl galactic2011-06-12/galactic2011-06-12.txt
**** galactic2011-06-12/galactic2011-06-12.txt
Artist: Galactic
Date: 2011/06/12

____Venue____
What Stage
Bonnaroo Music Festival
Manchester, TN
------
Track: 1    Name: Intro
Track: 2    Name: Who Took The Happiness
Track: 3    Name: I Don't Know But It Sure Is Funky (Corey)
Track: 4    Name: Never Called You Crazy (Corey)
Track: 5    Name: Keep Steppin
Track: 6    Name: Boe Money
Track: 7    Name: You Don't Know (corey)
Track: 8    Name: Night People (corey)
Track: 9    Name: Cineramascope
Track: 10    Name: Balkan Wedding
Track: 11    Name: Total destruction to your mind (corey)
Track: 12    Name: Corey Plead for Gulf Coast
Track: 13    Name: Heart of Steel (corey)
Track: 14    Name: Baker's dozen >
Track: 15    Name: Stanley Drums >
Track: 16    Name: Baker's Dozen
Track: 17    Name: How Many More Times (Corey)

____Comments____
Source: Soundfield SPS200 > Sound Devices 788t (24/48)
Location: FOB, DFC, 8.0 ft high

DSP: Sound Devices 788t > Wave Agent > Neundo 3 (sps200 szone vst, mixdown) > Wavelab 6 (fades, tracks, dither w/UV22 HR, resample) > FLAC v1.2.1 (L8) as build in TLH v2.6.0.(Build 168)

Recorded by: Daniel Kopp
Taped & Transferred by Daniel Kopp (taperhead@gmail.com)
Setlist help by featbase.net

*A Team Dirty South Recording*

//Please support artist by purchasing their albums, merch, and seeing their live shows. Also, tell your friends about them.//

One Set:

Galactic was joined entire performance by Corey Glover (Living Colour) and Corey Henry (Rebirth Brass Band)

Code: [Select]
$ perl ../parse-etree.pl gd1966-03-25.38/gd*txt
**** gd1966-03-25.38/gd1966-03-25.38.txt
Artist: Grateful Dead
Date: 1966/03/25

____Venue____
Troupers Hall
Los Angeles, CA
------
Track: 1    Name: Stealin'
Track: 2    Name: Jam
Track: 3    Name: Hey Little One
Track: 4    Name: Hog For You  Baby
Track: 5    Name: You Don't Have To Ask
Track: 6    Name: Cold Rain And Snow
Track: 7    Name: Next Time You See Me

____Comments____


S:MR>DAT>ZA2>CD

Comments:
Thayer Jennings: After You Don't Have To Ask,
Garcia says: "Okay, we'll be back in just a  few scant minutes."
& Lesh: "We're gonna take a little break now.
Don't  forget: on the 6th, 7th, 8th, 9th..."(cut).
DeadBase X: After Hey Little One -- Bob: "Hey Little One".
After Cold Rain And Snow -- Bobby: "That was Cold Rain And Snow".

Transcribed by Teddy GoodBear: Troupers Hall was the meeting room
for a retired actor's club in  Hollywood. The rent for the
gig could'nt. have been much. We did  everything ourselves,
all in two days. We plastered handbills all over  Hollywood.
Stage decor was a few lengths of paisley cloth purchased that 
afternoon at a fabric store. For a box office, we had a card
table and a  cigar box. Our not-quite-full house must have had
over a hundred people; and when the  night was over, our net
take was $75. At 2 o'clock in the morning, we went  to Cantor's
Deli on Fairfax and spent it all on dinner for everybody --  with
desert. -- Grateful Dead Family Album (page 43).
http://www.pompano.net/~goodbear/concert_art/03_25_66-1.html 
http://www.pompano.net/~goodbear/concert_art/03_25_66-2.html 
To listen to the show using "Real Audio":
http://sugarmegs.org/ram/gd3-25-66.ram 
http://www.deadlegs.com/tunes/1960/1966/32566/32566.ram 

Recordings:
30 SB. The complete list circulates in SBD.
The beginning of Stealin' is clipped with a significant portion
missing. The end of Next Time You See Me cuts with a significant
portion missing. Sometimes this  recording incorrectly circulates
as 01/??/66 or 01/22/66. To see a recording box label,
(as per Michael Wanger letter, see  07/??/66), from Golden State
Recorders: 
http://www.pompano.net/~goodbear/concert_art/03_25_66-3.html 

Contributors:
Jeff Tiedrich <jeff@tiedrich.com>, Thayer Jennings<htj@geocities.com>,
Teddy GoodBear <GoodBear@GeoCities.com>,
David  Sorochty <scarletfire@erols.com>,
Michael Wanger <vidkid@well.com>, 
Grateful Dead Family Album--p.43, DeadLegs <www.deadlegs.com>,
SugarMegs:  Mike Maloney <www.SugarMegs.org>.
Title: Re: Converting SHN files to FLAC files
Post by: kstuart on March 05, 2013, 02:07:55 pm
Looks good !

So, is this at the point where we can do some alpha testing with it ?
Title: Re: Converting SHN files to FLAC files
Post by: MrC on March 05, 2013, 10:01:51 pm
A little more to do first.

I've now finished:

- pulling/using band abbreviations from etree (http://wiki.etree.org/index.php?page=BandAbbreviations)
  so that a missing artist can be added from the abbreviation in the file name, and used for cross-checking
  an artist listed in the venue.
- collecting file names from the directory, parsing them, and correlating them to tracks
- better track parsing

Now the tough file (missing artist, no line breaks to demarcate venue, non-track info in between track list, longer than usual venue):

Code: [Select]
$ cat jg1962-06-11.txt
Sleepy Hollow Hog Stompers
Boar's Head Coffee House
Jewish Community Center
San Carlos, CA
6/11/1962
Master Audience (?) Reel (Scotch 150) >
playback on Technics RS 1520 @ 7-1/2 ips
no dolby > Barry Glassberg's first gen
reel (Scotch 207 recorded via Otari MX5050
@ 7-1/2 ips no dolby) > playback on Technics
RS1506 > Tascam DA-40 DAT > Tascam CD-RW700
master CD > CDs > HP 9350i extraction using
EAC > .shn encoding using mkwACT.  Source
discs from Jay Jurina, extraction and .shn
encoding by Joe Jupille.  Sector boundary
alignment confirmed using shntool.
Single Disc (18) 62:52
--Set I--
1. Tuning [0:23]
2. Run Mountain [4:50]
3. Billy Grimes The Rover [3:51]
4. Cannonball Blues [4:06]
5. Devilish Mary [4:05]
6. Buck Dancer's Choice [2:16
7. Little Birdie [3:56]
8. Sally Goodin' [2:32]
9. Hold The Woodpile Down [4:08]
--Set II--
10. Crow Black Chicken [4:05]
11. The Johnson Boys [3:57]
12. Shady Grove [5:22]
13. Hop High Ladies [3:34]
14. Sweet Sunny South [4:28]
15. All Go Hungry Hash House [3:37]
16. Man Of Constant Sorrow (1) [2:25]
17. Rabbit Chase [3:00]
18. Three Men Went A-Hunting [2:10]
Notes:
(1) Garcia acapella
Times given are for tracks, not songs.
According to Deadlists, the band consists of
Jerry Garcia (guitar and banjo), Marshall Leicester
(banjo and guitar), and Dick Arnold (fiddle). The
performance took place at the 'new' Boar's Head
Coffeehouse, relocated in the San Carlos Jewish
Community Center.  Deadlists also hypothesizes
that this is an audience recording, possibly with
onstage mics.

is parsed as:

Code: [Select]
$  ../../parse-etree.pl jg1962-06-11*.txt            
Base: jg1962-06-11, band abbr: 'jg', Artist from lookup: 'Jerry Garcia'
Artist: Jerry Garcia
Date: 1962/06/11

____Venue____
Sleepy Hollow Hog Stompers
Boar's Head Coffee House
Jewish Community Center
San Carlos, CA

____Tracks____

 1 Tuning [0:23]                            jg1962-06-11t01.shn
 2 Run Mountain [4:50]                      jg1962-06-11t02.shn
 3 Billy Grimes The Rover [3:51]            jg1962-06-11t03.shn
 4 Cannonball Blues [4:06]                  jg1962-06-11t04.shn
 5 Devilish Mary [4:05]                     jg1962-06-11t05.shn
 6 Buck Dancer's Choice [2:16               jg1962-06-11t06.shn
 7 Little Birdie [3:56]                     jg1962-06-11t07.shn
 8 Sally Goodin' [2:32]                     jg1962-06-11t08.shn
 9 Hold The Woodpile Down [4:08]            jg1962-06-11t09.shn
10 Crow Black Chicken [4:05]                jg1962-06-11t10.shn
11 The Johnson Boys [3:57]                  jg1962-06-11t11.shn
12 Shady Grove [5:22]                       jg1962-06-11t12.shn
13 Hop High Ladies [3:34]                   jg1962-06-11t13.shn
14 Sweet Sunny South [4:28]                 jg1962-06-11t14.shn
15 All Go Hungry Hash House [3:37]          jg1962-06-11t15.shn
16 Man Of Constant Sorrow (1) [2:25]        jg1962-06-11t16.shn
17 Rabbit Chase [3:00]                      jg1962-06-11t17.shn
18 Three Men Went A-Hunting [2:10]          jg1962-06-11t18.shn

____Comments____
Master Audience (?) Reel (Scotch 150) >
playback on Technics RS 1520 @ 7-1/2 ips
no dolby > Barry Glassberg's first gen
reel (Scotch 207 recorded via Otari MX5050
@ 7-1/2 ips no dolby) > playback on Technics
RS1506 > Tascam DA-40 DAT > Tascam CD-RW700
master CD > CDs > HP 9350i extraction using
EAC > .shn encoding using mkwACT.  Source
discs from Jay Jurina, extraction and .shn
encoding by Joe Jupille.  Sector boundary
alignment confirmed using shntool.
Single Disc (18) 62:52
--Set I--
--Set II--
Notes:
(1) Garcia acapella
Times given are for tracks, not songs.
According to Deadlists, the band consists of
Jerry Garcia (guitar and banjo), Marshall Leicester
(banjo and guitar), and Dick Arnold (fiddle). The
performance took place at the 'new' Boar's Head
Coffeehouse, relocated in the San Carlos Jewish
Community Center.  Deadlists also hypothesizes
that this is an audience recording, possibly with
onstage mics.
Title: Re: Converting SHN files to FLAC files
Post by: glynor on March 05, 2013, 10:04:44 pm
Seriously awesome.
Title: Re: Converting SHN files to FLAC files
Post by: Ekpen on March 05, 2013, 10:08:49 pm
MrC.
Thanks.
If my memory serves me well, the million record set someone (maybe) Glynor posted a while ago, contains SHN files. Can this be used to extract or convert the contents to FLac?

Thanks again,

Ekpen.
Title: Re: Converting SHN files to FLAC files
Post by: MrC on March 05, 2013, 10:27:10 pm
MrC.
Thanks.
If my memory serves me well, the million record set someone (maybe) Glynor posted a while ago, contains SHN files. Can this be used to extract or convert the contents to FLac?

Dunno.  Can you find the reference?
Title: Re: Converting SHN files to FLAC files
Post by: glynor on March 06, 2013, 06:05:43 am
MrC.
Thanks.
If my memory serves me well, the million record set someone (maybe) Glynor posted a while ago, contains SHN files. Can this be used to extract or convert the contents to FLac?

Thanks again,

Ekpen.

Not I.
Title: Re: Converting SHN files to FLAC files
Post by: Ekpen on March 06, 2013, 09:23:12 am
 A million record set-. I am referring to the one posted on March 11- 2011. I was able to locate it last night.
It has been moved to MC 15 area.
I thought the record set has SHN files. I took a look at one of the massive archives then, I saw SHN extension.
Ekpen.
Title: Re: Converting SHN files to FLAC files
Post by: kstuart on March 06, 2013, 10:33:45 am
A million record set-. I am referring to the one posted on March 11- 2011. I was able to locate it last night.
It has been moved to MC 15 area.
I thought the record set has SHN files. I took a look at one of the massive archives then, I saw SHN extension.
Ekpen.
On the basis of that information, I found the thread (indeed posted by glynor) at:

http://yabb.jriver.com/interact/index.php?topic=62892.0
Title: Re: Converting SHN files to FLAC files
Post by: Ekpen on March 07, 2013, 06:50:27 pm
Dunno.  Can you find the reference?
See my reply #23, if this may help.
Thanks.
Ekpen.
Title: Re: Converting SHN files to FLAC files
Post by: MrC on March 07, 2013, 06:54:52 pm
I saw it, but don't see any value in downloading 280GB of data.
Title: Re: Converting SHN files to FLAC files
Post by: kstuart on March 07, 2013, 07:07:31 pm
I just wanted to mention that it would be helpful if the parsing-text-file-and-writing-tags portion could be used separately from the converting-shn-files portion.

This is because there are a significant number of etree files that are already in flac format, but nevertheless have no tags, and have the information in the same text files (and the flac files are named with the same convention).

I remember the justification was something like "everyone has their own way of tagging files"... which, of course, doesn't consider that it would nevertheless be helpful to have the files originate with the information already in tags, and then users can still change it as they like...
Title: Re: Converting SHN files to FLAC files
Post by: glynor on March 07, 2013, 07:46:38 pm
On the basis of that information, I found the thread (indeed posted by glynor) at:

http://yabb.jriver.com/interact/index.php?topic=62892.0

Of course, I remember that now.  Didn't before.
Title: Re: Converting SHN files to FLAC files
Post by: MrC on March 07, 2013, 07:56:50 pm
I just wanted to mention that it would be helpful if the parsing-text-file-and-writing-tags portion could be used separately from the converting-shn-files portion.

Yup, already separate.
Title: Re: Converting SHN files to FLAC files
Post by: Ekpen on March 07, 2013, 08:05:03 pm
I saw it, but don't see any value in downloading 280GB of data.

You do not have to download all. If I remember correctly, there is a small set- sample within those sets "A" "B" etc, that may be of use to you.
I think those sets do contain SHN files, whether they are full blown  tracks or albums,  or not, I do not know.
Thanks.
Ekpen.
Title: Re: Converting SHN files to FLAC files
Post by: MrC on March 07, 2013, 10:25:02 pm
I now have working:

  - handle multi-disc sets
  - matching/stripping artist in Venue, using name found from etree

The file:

Code: [Select]
$ cat Garcia730504*txt
Jerry Garcia/Merl Saunders
5-4-1973
Homer's Warehouse, Palo Alto, CA
MSR>DAT3x>Delta DIO 2496>Soundforge>wav>flac
There are some imperfections that seem to be from the master reels.No adjustments made.


Disk 1 (Set 1) 
That's The Touch I Like
Expressway
Instrumental >
After Midnight >
It's No Use
That's All Right, Mama

Disk 2 (Set 2) 
It Takes A Lot To Laugh, It Takes A Train To Cry
Honey Chile
Lonely Avenue
Instrumental
Georgia On My Mind
The Night They Drove Old Dixie Down
How Sweet It Is

is parsed as:

Code: [Select]

$ perl   ../../parse-etree.pl  Garcia730504info.txt
____Artist____
Jerry Garcia/Merl Saunders

____Date____
1973/05/04

____Venue____
Homer's Warehouse, Palo Alto, CA
MSR>DAT3x>Delta DIO 2496>Soundforge>wav>flac
There are some imperfections that seem to be from the master reels.No adjustments made.

____Tracks____

Discs: 2

 1  1 That's The Touch I Like                  Garcia730504d1t1.shn
 1  2 Expressway                               Garcia730504d1t2.shn
 1  3 Instrumental >                           Garcia730504d1t3.shn
 1  4 After Midnight >                         Garcia730504d1t4.shn
 1  5 It's No Use                              Garcia730504d1t5.shn
 1  6 That's All Right, Mama                   Garcia730504d1t6.shn
 2  1 It Takes A Lot To Laugh, It Takes A Train To Cry
 2  2 Honey Chile                             
 2  3 Lonely Avenue                           
 2  4 Instrumental                             
 2  5 Georgia On My Mind                       
 2  6 The Night They Drove Old Dixie Down     
 2  7 How Sweet It Is

The venue in this sample file runs into the processing specs/comments.  I'll ignore that for now, and if there are more like this, I'll push such lines down into the comments.
Title: Re: Converting SHN files to FLAC files
Post by: MrC on March 08, 2013, 01:05:27 am
What do you want for the Album value:

  - The directory name?
  - Some portion of the venue?

Are there any other tags you want filled in (such as Keyword: Live, or Genre, or Media Sub Type)?
Title: Re: Converting SHN files to FLAC files
Post by: MrC on March 08, 2013, 02:29:30 am
Ok, I've now completed the .mpl creation, so tomorrow I'll be testing some importing.
Title: Re: Converting SHN files to FLAC files
Post by: kstuart on March 08, 2013, 11:30:48 am
What do you want for the Album value:

  - The directory name?
  - Some portion of the venue?

Are there any other tags you want filled in (such as Keyword: Live, or Genre, or Media Sub Type)?
This is just my personal reaction:

For live albums, the Artist and Date is the important factor.  The city is the aspect that gives it some specificity and color.  A lot of cities have a "Fox Theater" or a "Palladium", so it doesn't tell you much.

So

Miles Davis
1967-11-06
Paris

are the three things that characterize it.

So, for me:

Album would be "Live in Paris November 6, 1967",
Artist would be "Miles Davis",
Date would be "1967-11-06",
Venue would have as much information as found in the text file.

I think that

Keyword - Live

couldn't hurt and could be useful for many, and also

Comment = Converted from SHN set NNNNN

where NNNNN is a several digit number that may be in some filenames (ask me if this is not in any example I sent you).

Sounds good !
Title: Re: Converting SHN files to FLAC files
Post by: MrC on March 08, 2013, 11:42:58 am
Since I don't know the city with certainty, I can give you the venue portion for Album:

 Live in {venue} November 6, 1967

Since Venue is not a stock MC field, which field would you like for the captured venue?  Description?

Comments will contain everything not captured and stored elsewhere.

None of the files have the ##### number internally, but a couple of them in the file names I think:

   gd66-11-19.94106.txt
   gd1966-07-03.40.txt

Until I get a new version of MC, I'm at a standstill, since MC is crashing constantly with yesterday's build.
Title: Re: Converting SHN files to FLAC files
Post by: kstuart on March 08, 2013, 01:24:37 pm
I'm still on 129, because I am installing an OTA tuner and they don't work on any later version.

--

Yes, now that I think about it, there is no way for the parser to know what a "city" is. :)

--

The multi-digit number (called the "SHN ID") is a reference to a database at

http://db.etree.org/

Unfortunately, not every set of SHN files has them, otherwise you could just use the existing database fields from them.   Most GD sets have them, but shows of bands like "Dirty Dozen Brass Band" don't.

The number is helpful due to the presence of "upgrades".  For example, there are three different versions of 10-12-1968 and the most recent one is pitch-corrected.

--

For me personally, [Description] is where I put things that distinguish between different versions, so it would be good to have "converted from SHN files, ID # NNNNN" in that field.

And, it just occurred to me that [Places] would be good for the Venue information (trying to stay consistent with the field names, while using fields that are both searchable and saved).
Title: Re: Converting SHN files to FLAC files
Post by: MrC on March 08, 2013, 01:33:01 pm
Ok, good info.

The problem with Places and venue, is that venue will be multi-line, and a Large field works better, unless you want me to map multiple lines into a single line or a list of items in Places.  That's going to make it harder and uglier to edit.
Title: Re: Converting SHN files to FLAC files
Post by: MrC on March 08, 2013, 01:46:44 pm
We have some action!

From the following files and the generated foo.mpl:

Code: [Select]
$ ls
ddbb2010-12-11.textfile.txt  ddbb2010-12-11d01t02.flac    ddbb2010-12-11d01t04.flac    ddbb2010-12-11d01t06.flac
ddbb2010-12-11d01t01.flac    ddbb2010-12-11d01t03.flac    ddbb2010-12-11d01t05.flac    foo.mpl

$ cat *.txt
Dirty Dozen Brass Band
Warren Haynes Christmas Jam
Asheville Civic Center
Asheville, North Carolina
12/11/2010

Source: Schoeps CCM4V'S(din)>Lunatec V2>Benchmark AD2K>
Sound Devices 722 (24/48)
FOB/DFC/KFC/ZFC/AARP 40' From Stage, 6 1/2' High
DSP: Sound Devices 722>Sound Forge 9.0>CD Wave>flac(16)
Recorded By: Z-Man
Seeded By: Z-Man

Disc I

01 Introduction By Warren Haynes
02 Ain't Nothin' But A Party
03 Papa Was A Rolling Stone >
04 *#Spanish Moon
05 *@$^Superstitious
06 @$^+Use Me

* With Warren Haynes On Guitar And Vocals
# With Artemis Pyle On Percussion
@ With John Bell On Vocals
$ With Jojo On Keyboards
^ With Ron Holloway On Saxaphone
+ With Mike Barnes On Guitar

Flac Fingerprint

ddbb2010-12-11d01t01.flac:cff5e54a31ef32430a81e3144b722780
ddbb2010-12-11d01t02.flac:8b89092e90bb9545b4c132b49f4ea66d
ddbb2010-12-11d01t03.flac:bd9f90718a8ed579cfe775650699a169
ddbb2010-12-11d01t04.flac:479a48e7334dddc05864f333aa4781c7
ddbb2010-12-11d01t05.flac:fe76cb3c0cdec58f501f1d07bf07276c
ddbb2010-12-11d01t06.flac:d7f583a9e944076835288e2884b71021

Shntool
 
    length     expanded size    cdr  WAVE problems  fmt   ratio  filename
     1:21.01       14290796 B   ---   --   ---xx   flac  0.4037  ddbb2010-12-11d01t01.flac
    10:11.12      107808668 B   ---   --   ---xx   flac  0.5387  ddbb2010-12-11d01t02.flac
     7:35.40       80356124 B   ---   --   ---xx   flac  0.5394  ddbb2010-12-11d01t03.flac
     6:56.27       73445948 B   ---   --   ---xx   flac  0.5289  ddbb2010-12-11d01t04.flac
    10:12.53      108081500 B   ---   --   ---xx   flac  0.5402  ddbb2010-12-11d01t05.flac
     9:32.36      100985516 B   ---   --   ---xx   flac  0.5201  ddbb2010-12-11d01t06.flac
    45:49.19      484968552 B                            0.5298  (6 files)
Title: Re: Converting SHN files to FLAC files
Post by: kstuart on March 08, 2013, 04:08:31 pm
That looks awesome !  :o

--

I guess then put the venue information in Description.  Put the aforementioned stuff for Description first, then start a new line for the venue information.

(Using a sortable field makes it more easy to find information, in case you wanted Shuffle Play to do "only Avalon Ballroom recordings".   Using a saved-to-tags field means that one can re-do an MC18 installation from scratch.  I spent a lot of time yesterday entering info into MC18 database for cue-and-single-flac-file albums -- and none of that data gets saved back.  So, I realized I should avoid info that is not saved as tags, as much as possible - I will probably split those files some time later).
Title: Re: Converting SHN files to FLAC files
Post by: MrC on March 20, 2013, 08:49:32 pm
Ok, time to finish this up...

The conversion and mpl file writing code is now done.  I've successfully converted shn files to flac and written the mpl.  An imported mpl will pull in the corresponding flac files from the directory.  (see attached).

A new option (-c) will cause shn --> flac conversion.  When conversion occurs, filename suffixes in the mpl file are changed from .shn to .flac.  Without conversion (i.e. only writing an mpl file), suffixes remain as .shn.  This way, you can pull data from the info files to update your .shn files, and convert and pull tags for the flac files too.

The tool is run on a single directory of files.  It is given a single info file as an argument.  Flac and mpl files will be created in the same directory as the info and shn files.  Shn or flac files are not mandatory, but their presence helps better info file parsing (due to their typical naming conventions).

How do you want me to handle overwriting existing flac files?  How about a -f option to force overwrite, otherwise abort the conversion for the file?
Title: Re: Converting SHN files to FLAC files
Post by: MrC on March 20, 2013, 09:25:47 pm
@glynor - Do you have a couple of sample info files and directory listings or md5 files for me ?  I'd like a few more test samples.
Title: Re: Converting SHN files to FLAC files
Post by: glynor on March 21, 2013, 07:01:39 am
@glynor - Do you have a couple of sample info files and directory listings or md5 files for me ?  I'd like a few more test samples.

I'll dig through this weekend.

Sorry... I've been a stranger.  Work has been nutty.
Title: Re: Converting SHN files to FLAC files
Post by: kstuart on March 21, 2013, 11:30:54 am
Hmm... If this tool could convert CUE files to MPL files, it could kill off the biggest existing perpetual bug in MediaCenter....

although I just saw this just now:

" Fixed: When importing a FLAC with embedded CUE file, the last track could have the wrong duration. "
Title: Re: Converting SHN files to FLAC files
Post by: MrC on March 21, 2013, 11:33:54 am
Hmm... If this tool could convert CUE files to MPL files, it could kill off the biggest existing perpetual bug in MediaCenter....

Let's get the shn / info files done first...
Title: Re: Converting SHN files to FLAC files
Post by: MrC on March 21, 2013, 01:29:04 pm
Bump:

Quote
How do you want me to handle overwriting existing flac files?  How about a -f option to force overwrite, otherwise abort the conversion for the file?
Title: Re: Converting SHN files to FLAC files
Post by: kstuart on March 21, 2013, 01:58:20 pm
That sounds fine.   In general, it's always good to default to not writing, in case one triggers by accident.
Title: Re: Converting SHN files to FLAC files
Post by: MrC on March 21, 2013, 03:34:33 pm
Ok, time for you to test.  Here's what you need:

1. Download and install ActivePerl (or Cygwin tools - include perl and flac, and skip step 5)

2. Download and unzip the attached parse_etree.pl script and save it somewhere convenient.

3. shntool from http://www.etree.org/shnutils/shntool/
    http://www.etree.org/shnutils/shntool/dist/win32/shntool-3.0.10.zip

4. shorten from http://www.etree.org/shnutils/shorten/
    http://www.etree.org/shnutils/shorten/dist/win32/shorten-3.6.1.zip

5. flac from http://flac.sourceforge.net/
    http://sourceforge.net/projects/flac/files/flac-win/flac-1.2.1-win/flac-1.2.1-win.zip/download

6. Place the shntool, shorten, and flac executables (and dlls) in some folder that is in your PATH.  Or modify your PATH to include the directory where you store these files.

7. Open a command window and run the command.  The command syntax is:

   perl.exe parse_etree.pl [options] path\to\infofile

   Example:  perl.exe parse-etree.pl -cf  music\gd1966-07-03.39\gd1966-07-03.39.txt

The current options are:

      c=convert shn to flac (shn files are remain and are untouched)
      d=debug
      f=force overwrite of flac files
      m=don't write mpl
      v=verbose (output album info)

Without any options, only the MPL file will be written.  Use -c to cause conversion of .shn to .flac.  Use -f to force overwriting existing flac files.  If you want to see how the the info file is parsed, use just -v (so -vm will just output the parsing to the console and do nothing else).

The info file and .shn or .flac files must exist in the same directory.

8. In MC, use File > Import Playlist to import the newly created mpl file.  This will import the files referenced inside, and update the tags in MC.

If there are no shn files, but flac files already exist, you can still run parse_etree on an info file to create an mpl file, which can then be imported to add tags to existing flac files (which will also be imported).

If you have trouble, I'll need the info file, a directory listing of your files, and debug output wouldn't hurt.  Given the free-format of the info files, I'm sure there will be plenty of adjustments required.

Edit: attachment removed - see latest thread.
Title: Re: Converting SHN files to FLAC files
Post by: kstuart on March 21, 2013, 04:30:25 pm
The ActivePerl aspect is borderline, in that it is okay for individuals to use it for their own purposes.  Given their license wording, they would be okay with etree distributing it, but not okay with JRiver (non-commercial vs commercial).

So, it is good that you did not link to it explicitly.
Title: Re: Converting SHN files to FLAC files
Post by: MrC on March 21, 2013, 04:48:35 pm
There is nothing problematic with linking to the ActivePerl software download page or binary (in fact, I simply forgot to do so, as I have done so on other occasions).

I wrote a piece of code that might be of some use to you and other JRiver users.  Distribution of ancillary software, especially by other organizations, is entirely out of the realm and scope of this thread.
Title: Re: Converting SHN files to FLAC files
Post by: kstuart on March 21, 2013, 05:02:23 pm
So, let me see if I understand the options:

If you use -f , then the process will internally tag the flac files with the metadata.   So -cf (as in your example) means convert the shn files to flac, and then tag them.

Alternately, you can create an mpl file, and use that to import the metadata to MC18.  So, if you want to not actually convert the shn file, you could create mpl files, import them, and then see all the fields filled out in MC18 for the shn files.

Is that correct?

Title: Re: Converting SHN files to FLAC files
Post by: MrC on March 21, 2013, 05:21:03 pm
No, -f means Force Overwrite (of existing FLAC files).  Only meaningful with -c.

No file tagging is done - that is accomplished entirely via importing the MPL.

Correct about creating an MPL to import, which will set the tags for the corresponding SHN files.  Don't use -c in this case (-c means Convert); only the MPL file is created for the existing files.  SHN's are favored over FLACs when both exist.
Title: Re: Converting SHN files to FLAC files
Post by: kstuart on March 21, 2013, 05:35:16 pm
Okay, so the -f is just for cases where you rerun the same conversion twice (hopefully won't need that very often).

I'm trying it right now...

ActivePerl installs into a folder that it creates, and then it adds the path of that folder to PATH.   I put the three programs into System32.

However, it is looking like PATH doesn't work for the script, so you have to specify the absolute path if your command window is not run in the folder of the script.   And, in fact, the easiest way to convert a single folder is to use "Run Command Window Here", and then specify the absolute path of the script.

But going back to glynor's original concept - converting all the legacy shn file sets as a one time operation - it looks like we will want a batch file that traverses sub-folders and looks for .shn files and .txt files...
Title: Re: Converting SHN files to FLAC files
Post by: kstuart on March 21, 2013, 05:41:07 pm
I picked one folder randomly and here is what happened:

M:\Temp\gd73-02-26.sbd.kaplan.1208.sbeok.shnf>perl.exe c:\perl\parse-etree.pl -c
f  gd1973-02-26.1208.txt
Converting [gd73-02-26d3t01.shn] (25:18.39) --> [gd73-02-26d3t01.flac] : 100% OK

Converting [gd73-02-26d3t02.shn] (17:52.19) --> [gd73-02-26d3t02.flac] : 100% OK

Converting [gd73-02-26d3t03.shn] (7:18.07) --> [gd73-02-26d3t03.flac] : 100% OK
Converting [gd73-02-26d3t04.shn] (3:14.19) --> [gd73-02-26d3t04.flac] : 100% OK
Converting [gd73-02-26d3t05.shn] (6:35.40) --> [gd73-02-26d3t05.flac] : 100% OK
Converting [gd73-02-26d3t06.shn] (7:45.23) --> [gd73-02-26d3t06.flac] : 100% OK
Converting [gd73-02-26d3t07.shn] (3:00.07) --> [gd73-02-26d3t07.flac] : 100% OK
Converting [gd73-02-26d1t08.shn] (4:59.40) --> [gd73-02-26d1t08.flac] : 100% OK
Converting [gd73-02-26d1t09.shn] (7:06.16) --> [gd73-02-26d1t09.flac] : 100% OK
Converting [gd73-02-26d1t10.shn] (6:56.33) --> [gd73-02-26d1t10.flac] : 100% OK
Converting [gd73-02-26d1t11.shn] (3:22.04) --> [gd73-02-26d1t11.flac] : 100% OK
Converting [gd73-02-26d1t12.shn] (7:39.21) --> [gd73-02-26d1t12.flac] : 100% OK
Converting [gd73-02-26d1t13.shn] (4:15.38) --> [gd73-02-26d1t13.flac] : 100% OK
Converting [gd73-02-26d1t14.shn] (5:40.00) --> [gd73-02-26d1t14.flac] : 100% OK

So, it does seven tracks from Disk 3 and then seven tracks from Disk 1.

Here is the .txt:

Grateful Dead
February 26, 1973
Pershing Municipal Auditorium, Lincoln, NB

Source:  S:MR > cassette > DAT > CDR > EAC > SHN
Extraction and SHN encoding by Seth Kaplan
Thanks Seth for this show!

Info file and notes by G.S. Hamilton

Disc 1:
Set 1
1.  The Promised Land              3:05
2.  Loser                          6:47
3.  Jack Straw                      5:03
4.  Don't Ease Me In                3:34
5.  Box Of Rain                    4:53
6.  China Cat Sunflower >           6:47
7.  Jam                            0:42
8.  I Know You Rider                4:59
9.  Looks Like Rain                 7:06
10. Loose Lucy                      6:56
11. Beat It On Down The Line            3:22
12.  Row Jimmy                     7:39
13.  El Paso                       4:15
14.  Big Railroad Blues            5:39
Total time:                                 70:53


Disc 2:
Set 1, cont.
1.  Playing In The Band                     17:59
Set 2
2.  They Love Each Other                     5:38
3.  Big River                                4:23
4.  Tennessee Jed                            7:58
5.  Greatest Story Ever Told                 5:30
Total time:                                 41:30

Disc 3:
Set 2, cont.
1.  Dark Star >                             25:18
2.  Eyes Of The World >                     17:52
3.  Mississippi Half Step                    7:18
4.  Me And My Uncle                          3:14
5.  Not Fade Away >                          6:35
6.  Going Down The Road Feeling Bad >        7:45
7.  Not Fade Away                            3:00
Total time:                                 71:04

Notes:
D3t02 (Eyes): splice @ 3:25

==
Title: Re: Converting SHN files to FLAC files
Post by: kstuart on March 21, 2013, 06:08:56 pm
The FLAC files converted from SHN seem to play fine, and the tags look good - except that as shown above, the titles and durations are from the wrong tracks.   (Interesting that duration in the tags is a written text, rather than an audio duration, but that's an unrelated issue.)

I checked LiveShowTagger, and it has the Artist, Date, Venue and Setlist correct - except that it did not know to stop at "Notes:" and it lists " (Eyes):splice " as the title for 3/02, even though it had previously listed 3/02 as "Eyes Of The World >"...
Title: Re: Converting SHN files to FLAC files
Post by: MrC on March 21, 2013, 06:28:44 pm
Okay, so the -f is just for cases where you rerun the same conversion twice (hopefully won't need that very often).

I'm trying it right now...

ActivePerl installs into a folder that it creates, and then it adds the path of that folder to PATH.   I put the three programs into System32.

However, it is looking like PATH doesn't work for the script, so you have to specify the absolute path if your command window is not run in the folder of the script.   And, in fact, the easiest way to convert a single folder is to use "Run Command Window Here", and then specify the absolute path of the script.

But going back to glynor's original concept - converting all the legacy shn file sets as a one time operation - it looks like we will want a batch file that traverses sub-folders and looks for .shn files and .txt files...


Perl won't know where the script is, and perl won't search your path for it, so you have to specify it.  ActivePerl can be invoked by calling a script directly, but I think that requires a simple registry change.  Then, just placing the script in the path makes it executable just like any program.  Cygwin won't have this problem, since you can make the script directly executable.

Regarding converting an entire folder tree, I'd use the *nix find tool to do the recursion - it's a one-liner.

   find samples -name "*.txt" -type f -exec  parse-etree.pl -cf '{}'  \;

This is where *nix tools shine (so the advantage goes to Cygwin here).  (Actually, I did all the development on Mac OS X, and the same script just runs on the various platforms).

The difficulty with the above is that your info files are not a) well named, b) don't all end with .txt, and c) there may be multiple possible info files, so you'll have to know ahead of time how to specify the collection of info files so that find will find them.
Title: Re: Converting SHN files to FLAC files
Post by: MrC on March 21, 2013, 06:32:32 pm
The FLAC files converted from SHN seem to play fine, and the tags look good - except that as shown above, the titles and durations are from the wrong tracks.   (Interesting that duration in the tags is a written text, rather than an audio duration, but that's an unrelated issue.)

I checked LiveShowTagger, and it has the Artist, Date, Venue and Setlist correct - except that it did not know to stop at "Notes:" and it lists " (Eyes):splice " as the title for 3/02, even though it had previously listed 3/02 as "Eyes Of The World >"...

Are you saying the script got it wrong?  If so, I'll need to see the info file.

The durations aren't something the script is dealing with.
Title: Re: Converting SHN files to FLAC files
Post by: kstuart on March 21, 2013, 06:41:45 pm
Are you saying the script got it wrong?  If so, I'll need to see the info file.

The durations aren't something the script is dealing with.
See:

http://yabb.jriver.com/interact/index.php?topic=78216.msg539019#msg539019
Title: Re: Converting SHN files to FLAC files
Post by: MrC on March 21, 2013, 06:52:31 pm
See:

http://yabb.jriver.com/interact/index.php?topic=78216.msg539019#msg539019


I saw that.  Generally, pasted text does not work.  I think I've mentioned this before - it changes the line endings, and whitespace is destroyed.  I need the raw file(s).  The script relies on certain line patterns, and if something doesn't work, I don't want to chase my tail.
Title: Re: Converting SHN files to FLAC files
Post by: kstuart on March 21, 2013, 06:55:18 pm
I saw that.  Generally, pasted text does not work.  I think I've mentioned this before - it changes the line endings, and whitespace is destroyed.  I need the raw file(s).  The script relies on certain line patterns, and if something doesn't work, I don't want to chase my tail.
Okay, I'm happy to send you the files.... I was just assuming that there might be something about the visible syntax, irregardless of line endings...
Title: Re: Converting SHN files to FLAC files
Post by: MrC on March 21, 2013, 07:44:20 pm
Ok, I see the problem.  I didn't update the code in some sections to deal with multiple discs.  I'm working on a fix now...
Title: Re: Converting SHN files to FLAC files
Post by: MrC on March 21, 2013, 10:47:53 pm
Let's give version 1.1 a try (attached).  It runs against all your sample data.

Changes:
  - Multiple Discs are handled correctly, matching names with filenames, even when files are sparse
  - Automatically switches shntool executable name based on OS
  - Removes some track durations at the end of the track name
  - Miscellaneous bug fixes

Edit: attachment removed - see latest thread.
Title: Re: Converting SHN files to FLAC files
Post by: kstuart on March 21, 2013, 11:13:26 pm
You should remove the old version (or update the attachment)...
Title: Re: Converting SHN files to FLAC files
Post by: kstuart on March 21, 2013, 11:26:30 pm
So far so good on V1.1 run on the same files...

... I forgot to mention that Album came out as "gd1973"... the last mention in the thread was:

" Since I don't know the city with certainty, I can give you the venue portion for Album:

 Live in {venue} November 6, 1967 "

(Of course, gd1973 is not unique.)
Title: Re: Converting SHN files to FLAC files
Post by: MrC on March 22, 2013, 04:34:40 am
I've fixed this and a made venue and date parsing more robust.  I'll have a new version shortly...
Title: Re: Converting SHN files to FLAC files
Post by: kstuart on March 22, 2013, 12:00:11 pm
Two Qs on MPL files:

1) If you import an MPL file, and then change the Field Properties within MC18, and then do "Update Tags from Library", will it write the tags to a) the individual FLAC files, b) the MPL file, or c) both ?

2) Can an MPL file list "tracks" that are all within a single FLAC file (as can be done by cue files) ?
Title: Re: Converting SHN files to FLAC files
Post by: MrC on March 22, 2013, 12:08:17 pm
Two Qs on MPL files:

1) If you import an MPL file, and then change the Field Properties within MC18, and then do "Update Tags from Library", will it write the tags to a) the individual FLAC files, b) the MPL file, or c) both ?

2) Can an MPL file list "tracks" that are all within a single FLAC file (as can be done by cue files) ?


1. Individual files, not MPL.

2. I don't know how to do this, and have not tried.  My understanding is the the Filename field is what references the entry in MC.  So I imagine that importing an MPL with multiple records, all the same Filename, will just update the same entry over and over on a last-seen-wins basis.

I believe things like particles and cue entries are virtual in MC, and I don't know of any way to address them via MPL.
Title: Re: Converting SHN files to FLAC files
Post by: MrC on March 22, 2013, 04:56:17 pm
Here's version 1.2.

Changes:
  - Assigns Album tag as: Live in <first line of venue>, <formatted date>.  The formatted date can be configured.
  - More robust parsing of venue and dates with stricter filename vs. in-venue date correlation
  - Better debugging
  - Miscellaneous fixes and cleanups


Edit: attachment removed - see latest thread.
Title: Re: Converting SHN files to FLAC files
Post by: kstuart on March 22, 2013, 05:40:17 pm
Cool.  So what is the option for formatting the date ?
Title: Re: Converting SHN files to FLAC files
Post by: MrC on March 22, 2013, 05:52:06 pm
The formatting templates are described here:

   http://www.cplusplus.com/reference/ctime/strftime/ (http://www.cplusplus.com/reference/ctime/strftime/)

See the code:

    # format of date shown in Album as part of venue (strftime format)
    my $venue_date_fmt = '%B %d, %Y';

and change the $venue_date_fmt string using the codes from strftime.  Other characters can be inserted too:

    my $venue_date_fmt = 'On the fine %d day of %B in %Y';
Title: Re: Converting SHN files to FLAC files
Post by: kstuart on March 22, 2013, 06:17:23 pm
Aha, so you are just suggesting that the user just edit the script (probably many European users will want to reformat the date).   That makes sense, considering that one has to have a little tech savvy to run a commandline program.

Anyway, Version 1.2 worked fine on my test case, the Album, Date, Comment and Description are all great.
Title: Re: Converting SHN files to FLAC files
Post by: MrC on March 22, 2013, 06:34:55 pm
For now, that's the way you'll have to edit.  If there is more need, it can be a config file, command line option, or other.  Mileage first, tune-up next.
Title: Re: Converting SHN files to FLAC files
Post by: kstuart on March 22, 2013, 06:38:25 pm
Okay, just to clarify the use of the MPL file.

Since you mentioned that tag writing in MC18 does not write to the MPL file, this would indicate that it is being used here as an import mechanism (although I do see the Export Playlist command should be able to update the MPL file if desired).

So, to further automate the process, I think a chain of MCC commands need to be figured out, that would write the field information to the tags in the FLAC files (since this is part of the desired outcome).  I do see:

MCC_UPDATE_TAGS_FROM_DB

I guess that a batch file could use MCC to select the files listed in the MPL file, and do the above MCC command.  Is there a better way?

==

Also, assuming there are no bugs reported over the weekend, do you have any problem if I posted a link to this thread in a live show trading Forum or two, next week ?
Title: Re: Converting SHN files to FLAC files
Post by: kstuart on March 22, 2013, 07:02:46 pm
Auto-Import has a "Write file tags when analyzing audio, getting cover art and applying folder-based tags".  What we need here is "Write file tags when applying playlist-based tags".
Title: Re: Converting SHN files to FLAC files
Post by: MrC on March 22, 2013, 07:45:03 pm
I have found that just placing an MPL playlist in an auto-import folder that is configured to include the proper audio types and MPL playlists, the files will get automatically imported.  There's nothing you need to do.

Let's hold off on sharing the tool until we've worked out a few more kinks.
Title: Re: Converting SHN files to FLAC files
Post by: kstuart on March 22, 2013, 08:52:50 pm
I'm not referring to importing, I'm referring to writing the tags into the flac files themselves.   There are some easy workarounds, but it would be nice to have an elegant "solution". ;)
Title: Re: Converting SHN files to FLAC files
Post by: kstuart on March 22, 2013, 09:41:22 pm
Here's a CNBC Financial Reporter who will likely be interested in converting SHN files:

(https://pbs.twimg.com/media/BF0iRyvCcAA6XwE.png:large)
Title: Re: Converting SHN files to FLAC files
Post by: MrC on March 22, 2013, 10:22:34 pm
It is curious that an MPL will drag files in, as per auto-import rules, but not perform the configured analysis, etc.

Anyway, it is easy enough to use a smartlist or view that shows files with no tags or analysis info.
Title: Re: Converting SHN files to FLAC files
Post by: kstuart on March 23, 2013, 12:49:50 pm
Auto-Import has a "Write file tags when analyzing audio, getting cover art and applying folder-based tags".  I am assuming that if one does not analyze audio, get cover art, or apply any folder-based tags, then MC18 will not write file tags when importing.
Title: Re: Converting SHN files to FLAC files
Post by: MrC on March 24, 2013, 01:30:47 am
Here's version 1.3, which fixes the problem you mentioned in your email.  It also supports the non-standard track names.

Changes:
  - Fixed error that occurred when found filename date could not be matched against the in-venue date.
  - Support track names of the form [Tt]rack##.shn
Title: Re: Converting SHN files to FLAC files
Post by: kstuart on September 21, 2013, 09:15:12 pm
It is curious that an MPL will drag files in, as per auto-import rules, but not perform the configured analysis, etc.

Anyway, it is easy enough to use a smartlist or view that shows files with no tags or analysis info.
Okay, so I ran one of these conversions today, and it a) made FLAC files & b) made an MPL file.

Then I set Import to include MPL files, and then ran an Import, but the process did not apply the fields from the MPL file to the FLAC files.
Title: Re: Converting SHN files to FLAC files
Post by: MrC on September 21, 2013, 09:54:03 pm
I think you have to use File > Import Playlist.
Title: Re: Converting SHN files to FLAC files
Post by: kstuart on September 21, 2013, 10:39:34 pm
I think you have to use File > Import Playlist.
Ah, that worked perfectly, thanks much !