INTERACT FORUM
More => Old Versions => JRiver Media Center 22 for Windows => Topic started by: JimH on November 14, 2016, 10:43:19 am
-
MC22.0.41 added a new Listening Test, a comparison feature available from the Tools > Library Tools Menu. It allows you to listen to different formats of the same track, rate them, and then find out how good you are at distinguishing them from each other.
Here is how it works.
1. Find an audio track in your library.
2. Select it and choose Listening Test from the Library Tools menu.
3. MC will then convert the track to several new temporary files whose names are A, B, C, etc. and automatically put them in Playing Now.
4. Play them as much as you want, rate them with MC's star ratings.
5. When you click done, the results will be displayed.
6. When you press OK on the results page, the temporary files will be deleted. Your original file is not affected in any way.
-
Once you've tried it, please post your results. Most of mine were embarrassing. :-[ ;)
-
By different formats, do you mean different bit-depths and sampling, and lossy vs. non-lossy?
-
Right now, it's these:
FLAC 16/44.1
MP3 256Kbps
MP3 128K
MP3 64K
MP3 32K
We may change this in the future, but even now, it's a tough test to pass.
-
Right now, it's these:
FLAC 16/44.1
MP3 256Kbps
MP3 128K
MP3 64K
MP3 32K
We may change this in the future, but even now, it's a tough test to pass.
I Knew it!
-
The build required will be posted on this board in a few hours.
-
I've been wanting something like this for years to compare various formats.
I assume that for this test to be meaningful I will have to select a track with the highest possible bit depth as a starting point?
-
You could select anything MC can convert but it should be lossless at least.
-
Got version .41. Ran the test, a pop-up answered my above question (must be at least 44.1 kHz, 300 kb, 16 bit).
GREAT job with the info.
-
Won't convert my tracks. All my tracks are MP3 Extreme which means most are much less than 300 kbps.
Looks like a track has to be at least 300 (or lossless) as the popup stated.
-
You could rip a CD to FLAC, just to try it.
-
Coming next build we'll have this:
Fixed: The listening test wouldn't accept low or medium bitrate MP3 files.
-
Ahhh... YOU GUYS ROCK!!! ;D
This is going to be one great feature.
-
Crashed about 4/5 of way through three times using same file.
Windows 10 64 bit
Flac file was "Take Five"-24 bit, 176.4K version
attached a screenshot if that is helpful.
-
Crashed about 4/5 of way through three times using same file.
Windows 10 64 bit
Flac file was "Take Five"-24 bit, 176.4K version
attached a screenshot if that is helpful.
Coming next build:
Fixed: Doing a listening test with a really high sample rate file could lead to a crash.
Hopefully that solves the problem.
Thanks.
-
I did not test it, but does it include the original track also in the A, B, C, comparison list?
-
I did not test it, but does it include the original track also in the A, B, C, comparison list?
It does not include the original, but one of the copies is a lossless FLAC file so it's pretty much the same.
-
I was thinking of the people with HD audio (e.g. >16bit depth, >48kHz sample rate) who will certainly want to compare their HD originals versus the CD quality Flac transcode; or those who will want to compare their SACD / DSD rips versus the CD quality Flac. ( perhaps I'm thinking a bit ahead here.. )
So I would suggest to include a copy* of the original among the A/B/C selection. ( *I suggest to make a copy of the original so that the real original does not risk being lost by mistake.. )
-
Any chance for a traditional A-B comparison tool, that allows comparisons between 2 user chosen files with user controlled switch timing , with both a random and a user controlled switch. There have been requests for that over the years and it seems like a more general purpose tool. Foobar has an A-B comparison tool, but having to switch players changes the test conditions. I would suggest two modes - one with 2 user chosen files and one with 1 user chosen file with and without DSP enabled. An option to switch tracks either from the start or from the current playing position would be nice. No file copying necessary.
-
This is awesome! Can't wait to try it out!
Would it be possible to add an option where we could call the Format converter to generate 4-5 lossy files and manually choose our own conversion settings?
I could then use the external encoder option in MC.
I would personally love to blind test Lossless against Lame, Apple AAC (QAAC), fdk_aac and different bitrate settings I could manually change.
Or if easier just let us choose 4-5 files to choose from in a playlist (that I would convert myself manually in MC)...
-
Did anyone get all the tracks rated correctly?
-
I was thinking of the people with HD audio (e.g. >16bit depth, >48kHz sample rate) who will certainly want to compare their HD originals versus the CD quality Flac transcode ...
We'll try to add this.
-
But the huge problem with the Listening test is that it can change output settings that it should not. My e22 DAC can play DSD up to 256 and PCM up to 352,800, so in DSP studio I have Sample Rate set to "No change" for all rates. When the Listening test attempts to play the mp3 32 or 96 files MC tells me playback is not possible, and asks if it can change my output settings. For the sake of the test I said yes. When the test finished I checked the output settings and found that it had change EVERY sample rate other than 44.1 and 48 to 48. Obviously not required or desired.
I just tried with my e22 and had the same thing. MC is doing what it's supposed to. The e22 can't play anything below 44.1 PCM.
-
Crashed about 4/5 of way through three times using same file.
Windows 10 64 bit
Flac file was "Take Five"-24 bit, 176.4K version
attached a screenshot if that is helpful.
ALso crashed with a dsf file
-
All this is very interesting in theory. Most of us have specific comparisons in mind and would like to be able to select 2 or 3 original files and run a comparison. Personally, the formats I would be interested in comparing are all (well) above 16/44.1 and, possibly, in multichannel. Of course, that can run up against system resource limits.
-
I can definitely see myself spending a lot of hours on this if it allows for a bit more flexibility in the conversions...No need for the Wife to help out with blind tests anymore :)
Lossless vs Lossy
LAME vs AAC
Redbook vs Hi-Res
PCM Hi-Res vs DSD
And so on !!!
As Jim pointed out..I maybe in for a little surprise with the results I get...
-
Various conditions can affect this:
* State of mind (e.g. just read another article on election results ;D )
* Full tracks vs short section
* Headphones vs speakers
* Morning vs night
etc.
-
All this is very interesting in theory. Most of us have specific comparisons in mind and would like to be able to select 2 or 3 original files and run a comparison. Personally, the formats I would be interested in comparing are all (well) above 16/44.1 and, possibly, in multichannel. Of course, that can run up against system resource limits.
Agreed. I would use a tool that allowed me to select the tracks to compare, but doing pre-set low sample rates is not of much interest.
-
Tried three different files. All Lossless format, various bitrates. One jazz, one rock, and one classical.
Easily picked the FLAC file every time and the highest MP3 was 2nd all three times. Various success with the next three MP3s. Took a little time going back and forth through 5 formats. Started to get listening fatigue by the end.
It's a good start but would love an ability to compare HI-res vs 16 bit cd transfers. Also, if we could only do a one on one comparison from a choice it would be great.
-
Thanks. It's nice to see real data. It would be good to know how reliably you can pick the best and the worst format if you test 5 or 6 times.
-
All this is very interesting in theory. Most of us have specific comparisons in mind and would like to be able to select 2 or 3 original files and run a comparison. Personally, the formats I would be interested in comparing are all (well) above 16/44.1 and, possibly, in multichannel. Of course, that can run up against system resource limits.
There is a new build here:
https://yabb.jriver.com/interact/index.php/topic,107927.0.html
It uses the original file as one of the test files.
-
There is a new build here:
https://yabb.jriver.com/interact/index.php/topic,107927.0.html
It uses the original file as one of the test files.
And the options are there for the others?
-
At this time, it only converts to lower resolution formats. Is that what you're asking about?
-
I'll fess up on my results - I've used a couple of well listened to tracks on a pair of Grado SR60....
This is very informative. Another subjective listening test over a few different tracks. I can easily pick the MP3 32 and 96.... maybe the 128 (but I'm probably kidding myself), and certainly can not differentiate the 256 and FLAC tracks. So for my ears it looks like the stock std CD / BD lossless Formats are fine and already deliver a sound quality better than I can discern. That said I do hear a different between differently mastered albums with the later HD tracks much more pleasing (but this presumable is more to do with the re-master rather than the formats being used).
-
I can (sometimes) tell the difference between 256 and FLAC on tracks with lots of loud low bass on systems that can reproduce it. I think that's because one of the ways that mp3 compresses is by attenuating very high and very low frequencies. There's definitely one track I have with some some loud 25Hz rumbles where I can get 4 out of 5 right every time and occasionally all 5 (I can just listen for volume/distortion and it's pretty obvious).
But it's kind of a synthetic test. For actual, you know, music, I can't do anywhere near that well.
Also a small "bug" If you hover over the tracks while you're rating them, the tool tip will pop up and show you the file name, which will tell you whether it's an mp3 or a flac. The file names should either be less obvious, or the tool tip should be disabled I would think.
-
At this time, it only converts to lower resolution formats. Is that what you're asking about?
Too bad. I want to compare specific formats, e.g., DSD vs. DXD or DSD64 vs. DSD128 vs. DSD256, and nothing below 16/44.1/
-
Too bad. I want to compare specific formats, e.g., DSD vs. DXD or DSD64 vs. DSD128 vs. DSD256, and nothing below 16/44.1/
We may do that in the future, but what happens if you use it now? Can you reliably identify the best format and the worst?
-
I want a way to compare APE and FLAC files because to my ears, FLAC always sounds a little flat by comparison. APE files are just perfectly true to the original and FLAC is just lacking that little edge.
Ha!
-
Also a small "bug" If you hover over the tracks while you're rating them, the tool tip will pop up and show you the file name, which will tell you whether it's an mp3 or a flac. The file names should either be less obvious, or the tool tip should be disabled I would think.
The tooltip was supposed to be disabled in the last build or two. What build are you using?
-
I want a way to compare APE and FLAC files because to my ears, FLAC always sounds a little flat by comparison. APE files are just perfectly true to the original and FLAC is just lacking that little edge.
Ha!
Isn't the track fully decompressed before it plays in MC? Just wondering how that would be possible...
-
The tooltip was supposed to be disabled in the last build or two. What build are you using?
I'm using .41, which is the one where tooltips are supposed to be off, but they weren't for me.
-
On Windows?
-
We may do that in the future, but what happens if you use it now? Can you reliably identify the best format and the worst?
From long history, I have no interest in MP3 so testing it is, imho, a waste of my time.
-
From long history, I have no interest in MP3 so testing it is, imho, a waste of my time.
I'm not pushing MP3 at all. I just want to know two things.
Is the test working for most people?
At what point can most people reliable and consistently hear a difference?
If you don't want to try it, that's fine. I do expect that we will expand it or add different tests.
-
Is the test working for most people?
As I said above, A-B testing with the user selecting the tracks would be of use, but I have no interest in the current implementation.
In my house, if I put on an MP3 my wife walks in and says "what's that?. She definitely can hear the difference. That's all the test I need.
-
On Windows?
Yes .41 on windows.
-
I have tried quite a few 24/192 FLAC files , all with the same result.
MC crashes when the progress bar gets to the end.
Standard 16/44.1 FLAC files work fine. I always make the two lower rate MP3s and the original. The other two bitrates (256 & 128) sound the same to me.
-
Are you using 22.0.41?
https://yabb.jriver.com/interact/index.php/topic,107927.0.html
-
Too bad. I want to compare specific formats, e.g., DSD vs. DXD or DSD64 vs. DSD128 vs. DSD256, and nothing below 16/44.1/
If you want to do that with someone else's files (and in a non-automated way), you can go to:
http://www.2l.no/hires/index.html
for free downloads.
-
If you want to do that with someone else's files (and in a non-automated way), you can go to:
http://www.2l.no/hires/index.html
for free downloads.
Yes, I know but I have many others as well. It is the automated, blind testing that I seek.
-
Nice feature. Little embarrassed I picked MP3 128 as my second favorite on one song but at least I did pick flac as the best. On another I picked flac as the third best, lol. Should get rid of the MP3 32, sounds horrible and is easy to spot. Maybe replace it with MP3 160 or an even harder challenge would be MP3 320.
;)
-
Should get rid of the MP3 32, sounds horrible and is easy to spot.
32Kbps is there so everyone is a winner on something.
-
Are you using 22.0.41?
https://yabb.jriver.com/interact/index.php/topic,107927.0.html
Yes I do.
-
If JRiver would support letting the user choose the A file and the B file separately, then users can compare any difference in processing they want. This is the most flexible, and it would make the "Listening Test" feature a true AB compare. For me, it is only important to compare WAV files of the same sample rate. If I want to compare against a lossy compressed format, I can decode it to WAV first. In fact, I suggest that a true AB compare be limited to files with the same sample rate at the feature introduction for ease of implementation.
I do professional music mastering, and I need this to compare different versions of my processing. I bought MC22 because the "Plans for MC22" thread says that the "AB Comparison for listening tests" feature is included.
While cross fading between A and B tracks avoids a possible click, I find that it interferes with my ability to hear the difference between A and B. Differences in spacial ambience are much easier to hear when the transition between A and B is as quick as possible. If MC users insist on cross fading, please make it a defeatable option.
Please also ensure that MC supports letting the user train on the A and B files manually (sighted) for as long as they want before starting the blind testing phase. foobar2000 was smart to include this in it's ABX comparator.
-
If JRiver would support letting the user choose the A file and the B file separately, then users can compare any difference in processing they want.
Agreed.
Did people request the ability to compare lower resolutions? I record music and often want to compare microphones, preamps, conversion techniques, mic patterns, locations, etc. Those are different files. I've never said to myself, "Gee, I wish I could compare a 128k MP3 to a FLAC file." Maybe other people did, but certainly not the limitations that I anticipated for an AB test. When is version 2 due? :)
-
Agreed.
Did people request the ability to compare lower resolutions? I record music and often want to compare microphones, preamps, conversion techniques, mic patterns, locations, etc. Those are different files. I've never said to myself, "Gee, I wish I could compare a 128k MP3 to a FLAC file." Maybe other people did, but certainly not the limitations that I anticipated for an AB test. When is version 2 due? :)
As I said above, we may extend this feature, but for now, let's see whether you and others can successfully tell the difference consistently between FLAC and 256Kbps MP3. So far, the record isn't clear.
-
It should probably be called a hearing test or equipment test because it doesn't seem to have the listening features people are looking for.
-
but for now, let's see whether you and others can successfully tell the difference consistently between FLAC and 256Kbps MP3. So far, the record isn't clear.
Dear Jim, I don't think it should be your role (even if you are the CEO) to try to "prove" that people can or cannot hear a difference between Flac and Mp3. And even if you do "prove" something, your role is certainly not to rant about it. On the contrary I think your role is to deliver the best dang Media Player that money can buy, and deliver features that give your customers the most amount of fun.
-
We can certainly remove the feature if people are offended. There are plenty of other things we could be doing.
-
It should probably be called a hearing test or equipment test because it doesn't seem to have the listening features people are looking for.
Sorry, I should have said that I like the feature. The only thing I would change would be that I would like to switch between files as they are playing and without the next track starting over.
-
ive never seen *anyone* reliably identify 320kbps(or 256 for that matter) vs redbook/lossless
great feature, yall! thanks!
-
Cross Fade should be turned off, then turned back on after your done with the test..
I was getting a crash(with warning) with 22.0.43, Sometimes, not sure yet what may trigger.
next time it does it i will take a print screen.
-
ive never seen *anyone* reliably identify 320kbps(or 256 for that matter) vs redbook/lossless
I've done it many times. Not with the JRiver tool, mind you, as it's not yet a suitable tool for such testing.
-
If you compare two files, there is a 50% chance you'll get it right.
Can you reliably rank the FLAC above a 256Kbps MP3, five times in a row?
For all the complaints about what we did, I'm not seeing many reports of real tests. That suggests that people may not be reporting the results. Or that there is a complete lack of interest.
-
Or that there is a complete lack of interest.
Yes.
By and large the result of comparisons is uninteresting. Done some in the past and now just want to play the music.
There is room for some mildly interesting comparisons. I'd be interested to see the results of a properly conducted, statistically significant comparison of HD audio with the same source properly converted to CD audio quality to find out if the difference was actually detectable (it's almost certainly insignificant). I haven't seen any so far - too much vested interest on the HD side and not enough interest on the rational side.
-
I've removed some off topic posts.
If you have suggestions on improving this feature, please use this thread. If you want a different feature, please use another thread. I realize that you may find this a difficult distinction.
-
Split mwillems post here:
https://yabb.jriver.com/interact/index.php/topic,108131.0.html
-
If you have suggestions on improving this feature, please use this thread.
My suggestion would be a way of testing a file playing normally against the same file with the computer driven close to maximum processing etc.. It is often claimed by a section of the audiophile community that "reducing processing" or some such improves sound quality.
Add to that comparing a file being played back from memory to the same file being read from disc.
-
If you compare two files, there is a 50% chance you'll get it right.
Can you reliably rank the FLAC above a 256Kbps MP3, five times in a row?
Me? Yes. But as I've said before, not with the MC Listening Test as the tool gets in the way of the process. 320 vs FLAC is the real challenge and requires quite a bit of concentration to do, but it's doable and to a statistically significant degree.
At lower bitrates there are obvious "tinklies" (high frequency distortion) and other artefacts that give mp3 away. At 320, they are mostly absent, but tinklies can still be there in certain instruments, eg tambourines/chimes/cymbals, ie the high-frequency instruments. But the giveaway differences usually come down to things like the air between the instruments, or the definition of the instruments themselves, with the mp3 smearing the boundaries ever so slightly, presenting a more homogenous sound, and FLAC giving the accurate presentation. Often subtle, but always there.
The tool is fine for showing that mp3 can be "good enough" for casual listening, eg sat at a computer typing this message. Maybe that's all you are trying to do.
-
Useful feature IMHO. I only did one test run using an 8 minute Bruch violin CD track (so 40 minutes total test time), where I listed to each track just once before ranking. Clearly I should do more test runs with different recordings on different audio systems. The test was run on my desktop using the motherboard Realtek audio chip and Pioneer BR41 speakers, alternative options are to replace the Realtek with an ODAC USB DAC, and also run tests on my more expensive (higher quality?) home theater audio system to see if I can repeatedly, accurately discern differences in all three equipment sets, just the more expensive system or none at all.
In my one run through, my 58 year old ears detected the the 32k and 64k accurately but didn't rank the 128k, 256k and Flac correctly. For reference, I have no history of exposing my ears to loud concert music but online audio tests played back through the first set of equipment suggest a high end frequency limit of ~13-14k. I deliberately chose a relatively long audio test segment to help reduce/avoid audio memory since it seems to me the value in a test like this is to see what one perceives in absolute terms without a relatively recent audio reference to compare to. Comparing relatively short audio snippets may enable me to be more accurate at the higher bit-rates but its not how I listen to (and enjoy) music.
Interestingly my perception of 128k classical audio streams has been that they sound "compressed" and "lacking highs" when selecting either higher bit rate streams or playing back my CD rips, but there are so many other differences when doing so that its actually meaningless, so I'll be interested in more rigorously testing the ability to determine between 128k vs 256/Flac .
-
I moved a few posts. Please use this thread for general discussion:
https://yabb.jriver.com/interact/index.php/topic,108131.0.html
-
Does it works with a CD track?
-
Great Idea and a good test thanks for the effort. Interesting to try it out so keep up the good work.
Just did two quick tests, first one i got all 5 in the correct order....so was confident....
Second test got the flac and the 32 correct the other 3 were the wrong order!
Its late and need to try it out properly when i'm not tired.
The flac stands out by a long way from the others and is obvious as the best quality and 32 is so obviously the worse.
and not to state the obvious but some systems wont be able to pick up the differences and some will, so it maybe relevant for people to state what equipment the test was ran on.
I am using an audiolab dac/pre amp, bryston poweramps and pmc ob1 speakers.
-
Great report. Thanks.
-
Just wanted to say that I like this option! I have tried it once. I am embarassed to say that I only got the 32 correct! Ugh!! But I will try again and see if I can replicate my results. I hope NOT!
Best Regards,
Blu99Zoomer
-
Thank you for the Listen Test. For my first use of this feature I utilized the track Freddie Freeloader from Kind of Blue by Miles Davis (which is a 192 KHz 24 bit FLAC version purchased from HD Tracks). Not withstanding my need to check the Hardware Buffer box in ASIO4ALL in order to eliminate issues associated with 192 KHz playback in my system (for all other sample rates the box must remain unchecked), I was able to clearly discern all of the lower quality MP3s from the FLAC versions. It does require active listening, but you can really hear the difference. This result is a particularly satisfying confirmation that my decision to build my computer around my HT Omega Claro Halo (outfitted with version 4 Burson Supreme Sound Op Amps) was indeed a good one.
-
Nice test Jim, I enjoyed it.
Ross
-
Copied this from another thread:
Well I think that it's not impossible to hear a difference when a component is swapped.
But on topic: would it be possible to add functionality to this feature (or implement it as a new one with a different Name) in Order to compare two manual selected tracks from the library? If so, Media Center should try to allign those tracks.
Another related thread:
http://yabb.jriver.com/interact/index.php/topic,109305.0.html
;D
-
This is a great feature! Sounds funny, but this might be the reason of updating from MC21... :o
-
This is an interesting feature, and I'd like to see it expanded so it could start with higher resolutions than CD quality. I agree with Kal on that.
mp3 is a dead issue to me, and I suspect to many audiophiles. Storage is cheap and plentiful, so why make lossy copies of material? I do use mp3 in my car, whose media player won't do flac, and it's perfectly fine for that. I know that high-rate mp3 can sound good. But so what? I have no need for it.
But what about sound-quality comparisons of lossless material at and above CD quality? Do the 24-bit, 44.1 kHz tracks in my library sound worse if dithered to 16 bits? What happens to quad-rate tracks when downsampled to double rate? Single rate?
-
Did you try the test? Could you post your results?
-
I am not to surprised by my results. I have done blind tests in the past on websites with songs I am not very familiar with and it is harder to tell. But I have never had a blind test where I could pick the song. So I chose Comfortably Numb by Pink Floyd. Original was a 24/96 high quality vinyl rip. I chose to do all formats. Keep in mind it is hard for me to rate this song low. So Most of my ratings are 3 or 4 stars as an acceptable version if needed.
But I chose
5 Stars for DSD
4 Stars for FLAC
4 Stars for 256 MP3
3 Stars for 128 MP3
3 Stars for 96 MP3
1 Star for 32 MP3
I may try again with different tracks to see if I can still pick that well. I do consider myself to have very good listening ears though.
Thanks a lot for this feature. It was nice to be able to justify the hard drive space.