INTERACT FORUM

Please login or register.

Login with username, password and session length
Advanced search  
Pages: [1]   Go Down

Author Topic: bit for bit comparison of files (APE vs FLAC)  (Read 10468 times)

enigman

  • Regular Member
  • Junior Woodchuck
  • **
  • Posts: 95
  • thinking...
bit for bit comparison of files (APE vs FLAC)
« on: June 14, 2006, 01:37:31 pm »

I recently converted my entire library from APE to FLAC.  However, something clearly went wrong during this process, and some (but not all) of the Flac files have random problems with them, compression artificacts, clicks and pops, etc. 

No idea what caused this, but it only seems to have affected a handful of files, and on re-encode they seem fine.  But of course, this has me nervous now, and I'd like to go through and do a bit-for-bit comparison of the uncompressed output of all my files, since I thankfully still have the original APEs. 

Is there any way to do this within MC or can anway recommend an external program?  One issue: when I used "convert format" I moved the FLAC files to a new drive, and MC redid the folder structure, so matching up the old files with the new ones could be a big pain unless the comparison program is smart (I think the actual filenames should still be the same with the exception of the extension change)

Thanks!
Logged

Matt

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 42048
  • Shoes gone again!
Re: bit for bit comparison of files (APE vs FLAC)
« Reply #1 on: June 14, 2006, 01:52:31 pm »

This could be bad memory or a failing chipset.

APE has strong error detection built in, but I'm not sure how you could check the FLAC files.
Logged
Matt Ashland, JRiver Media Center

enigman

  • Regular Member
  • Junior Woodchuck
  • **
  • Posts: 95
  • thinking...
Re: bit for bit comparison of files (APE vs FLAC)
« Reply #2 on: June 14, 2006, 02:01:08 pm »

it's a brand new machine, so I'm hoping it's not that  :o

The external hard drive the original files were on is a bit flakey (randomly unmounts from time to time), so that could be the problem.  I've also noticed that the problem seemed to occur when I was ripping a CD at the same time as I was running the file conversion, but since the conversion process took about a week (on and off), I don't really know which specific files could have been affected. 

Again, my hope is that there's a tool out there which will decode the APE, decode the FLAC, and compare the results.  The trick of course is going to be syncing everything up given that files have moved.  All suggestions are most welcome!
Logged

hit_ny

  • Citizen of the Universe
  • *****
  • Posts: 3310
  • nothing more to say...
Re: bit for bit comparison of files (APE vs FLAC)
« Reply #3 on: June 14, 2006, 03:02:37 pm »

I recently converted my entire library from APE to FLAC.
What's to gain ?

going from one lossless codec ....to another  ?
Logged

enigman

  • Regular Member
  • Junior Woodchuck
  • **
  • Posts: 95
  • thinking...
Re: bit for bit comparison of files (APE vs FLAC)
« Reply #4 on: June 14, 2006, 04:21:37 pm »

Flac is supported much better outside of MC than APE.  APE's great otherwise, and better than FLAC in some ways, but file support was the deciding factor.  Too bad the conversion bugged out!
Logged

hit_ny

  • Citizen of the Universe
  • *****
  • Posts: 3310
  • nothing more to say...
Re: bit for bit comparison of files (APE vs FLAC)
« Reply #5 on: June 14, 2006, 04:51:08 pm »

Again, my hope is that there's a tool out there which will decode the APE, decode the FLAC, and compare the results.  The trick of course is going to be syncing everything up given that files have moved.  All suggestions are most welcome!
I'm not aware of a tool that can do this but a complicated batchfile just might. Even then it will be quite slow as each file needs to be converted to WAV twice from both formats and then compared.

Your experience has highlighted  an important point, the choice of lossless codec matters as even though its possible to losslessly convert to another lossless format, its much harder to verify the ouput. And if its a big collection, this could turn out to be even longer. Of course if you got 2GB+ of RAM to play with, things might be more feasible.

Now if (and its a pretty big IF), APE & FLAC saved a checksum of the orignal WAV they were made from, the comparison just got a lot faster. This could also serve as an instant check that the WAV if extracted from either format was ok, a mismatch implies the compressed file is corrupt in some way.

Unless you find such a tool, it seems wiser at this point to stick with APE (still have the orignal APE files) until its possible to verify the conversion.
Logged

Alex B

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 10121
  • The Cosmic Bird
Re: bit for bit comparison of files (APE vs FLAC)
« Reply #6 on: June 14, 2006, 05:22:26 pm »

It's the revenge of the monkey.   ;D


Seriously speaking, the only usable solution I know is to compare the files manually with Foobar's bit compare tool (a file pair at a time).

In any case you would need to decompress all files (FLAC & APE) during the compare process. After that you would still need to encode the replacement files.

I would connect an extra HD internally and reconvert all files. Decoding a file and encoding a new file is not much slower than decoding two files and comparing the decoded data especially if a fast and reliable internal drive can be used.

Also, I would keep the original APE library as a backup archive (at least for now). My second backup archive could be in FLAC format.
Logged
The Cosmic Bird - a triple merger of galaxies: http://eso.org/public/news/eso0755

Alex B

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 10121
  • The Cosmic Bird
Re: bit for bit comparison of files (APE vs FLAC)
« Reply #7 on: June 14, 2006, 05:48:15 pm »

...The external hard drive the original files were on is a bit flakey (randomly unmounts from time to time)...

I reread your post and noticed that your original files are on that possibly broken external drive.

That is quite serious. A hard drive must be 100% reliable all the time. I would immediately buy a couple of new drives and try to copy the data to a safer place(s).

BTW, do you use the latest FLAC plugins from scthom's site?
http://sourceforge.net/project/showfiles.php?group_id=161914
Logged
The Cosmic Bird - a triple merger of galaxies: http://eso.org/public/news/eso0755

JimH

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 71656
  • Where did I put my teeth?
Re: bit for bit comparison of files (APE vs FLAC)
« Reply #8 on: June 26, 2006, 12:41:09 pm »

See this post for the happy ending story by enigman:

http://yabb.jriver.com/interact/index.php?board=3;topic=34330.4#msg234428
Logged

hit_ny

  • Citizen of the Universe
  • *****
  • Posts: 3310
  • nothing more to say...
Re: bit for bit comparison of files (APE vs FLAC)
« Reply #9 on: June 26, 2006, 01:50:42 pm »

After some detailed manually analysis (using FooBar2000 across a bunch of random files and listening to a lot my library), I discovered that the problem was a lot less severe than I had originally thought.  There were only 2 or 3 albums that got messed up, due to a hardware issue when I was ripping while also converting.  This was then compounded by buggy audio driver that was causing skips in the S/PDIF output, leading me to suspect that the files themselves were messed up.

All is well now, and while I may have liked some sort of verification step to confirm that things went 100% according to plan, the end result was still quite positive.  FLAC is definitely the way to go - no disrespect to the monkey - in the abstract it's a great format, but it's just not as widely supported.
You were fortunate in that the problem was less severe than perceived but nonetheless your post highlighted an important issue.

Are there any tools to check the integrity of APE or FLAC files ?

Assuming the original APE files are correct ie no skips etc or integrity errors, would it then just be a question of checking the integrity of the resulting FLAC files and safely assume all went well ?

In the mp3 world there are such tools and its very fast to find problematic files. These are usually caused by a hardware problem somewhere (dodgy cable or memory, bad HDD)  in that not all bits are written accurately.
Logged

scthom

  • Citizen of the Universe
  • *****
  • Posts: 621
Re: bit for bit comparison of files (APE vs FLAC)
« Reply #10 on: June 28, 2006, 12:10:40 am »

I would use the command-line flac to batch check for errors in files.
Logged
Pages: [1]   Go Up