Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[mwillems]

So I'm relatively new to PC video; until a few months ago almost all of my video watching was more or less direct from DVD or Blu Ray discs, but I've recently been watching more video directly from the PC, and have some questions about what can be done about the enormous file sizes.  Specifically, I'm curious about what folks' think about using MC's convert format function (or another similar utility) to save Hard Drive space.

So here are some of the issues as I understand them (please correct me if any of these premisses are incorrect):  

1) Most commercial video is compressed using a lossy compression method, so any attempt to reencode it will necessarily degrade quality/result in information loss to some extent, but may or may not be noticeable.

2) The accompanying audio tracks can be lossy or lossless depending on the compression algorithm used, and lossless audio tracks can be recompressed to another more efficient lossless compression method.

My approach so far has been to take what steps I can to reduce filesizes losslessly, by using makemkv to snip out special features I'm not interested in and using makemkv's FLAC encoder to re-compress videos with lossless audio tracks.  Using those two methods I can typically shave off 10% to 30% as compared to the original file size  (FLAC recompression alone has been getting me gains of 5% to 10%), but that still leaves very large files.  

What I'm wondering about is, does anyone have any recommendations for format conversion/reencoding settings that would produce additional size reductions of at least 15%, but without obvious quality degradation, either using MC's convert format tool, or handbrake (or something similar)?  

I recognize that the question is kind of like "how long is a piece of string" in that everyone will have a different idea about what "obvious quality degradation" looks like, but I'd be curious to hear what settings/tools folks here use to produce results that they themselves are happy with or find visually indistinguishable from the original.   There are a lot of folks on this forum who are have a very sophisticated approach to video, and I'd like to get a calibration on what looks good to other people.  

I've tried a few convert format options in MC and a few handbrake settings, but I've been unhappy with the results so far.  Re-encoding with high quality settings takes a while on my PC, so blind trial and error is inconvenient and frustrating.  I guess I'm just looking for some places to start that are more likely to produce good results than just blundering around.  Thanks in advance for any help.

[6233638]

Handbrake is (mostly) a GUI for x264, which is the best H.264 encoder out there.
Depending on what display you are using (size, resolution, and type) then re-encoding may produce acceptable results.
 
I find that a 50GB disc usually has 25-30GB reserved for the main feature (sometimes less) and many people seem happy with the quality achieved by compressing that to about 10-15GB via Handbrake.
 
Personally, I do notice this loss of quality - especially if you will be upscaling it (e.g. on a 4K display) and the process is far too time consuming/CPU intensive for it to be worthwhile in my opinion.

Play around with the CRF settings until you find something which produces acceptable results.
Turning up the encoder settings from the defaults can improve quality for any given size, but greatly increase encoding time.
Any time I tried this though, I ended up turning the encoder settings up as high as they went, and after settling on a CRF setting which looked good, the file size was almost the same if not larger than the original.
 
 
These days I will either rip to ISO if I don't have any time to spend looking over a disc, or use MakeMKV to extract the main title.
 
Usually I end up converting to MKV fairly shortly after - most discs only take a minute or so of playback/seeking to determine whether they will be easy or "complex" to rip.
E.g. lots of different titles to sort through, finding whether they contain non-English dialog/subs etc.

I use this as a preset (save it as a .mmcp.xml file in the MakeMKV directory) which greatly simplifies things - though it does still require some intervention.

Code: (Custom Profile) [Select]

<?xml version="1.0" encoding="utf-8"?>
<profile>
    <!-- profile name - Default -->
    <name lang="eng">Custom</name>

    <!-- Common MKV flags -->
    <mkvSettings
        ignoreForcedSubtitlesFlag="true"
        useISO639Type2T="false"
        setFirstAudioTrackAsDefault="true"
        setFirstSubtitleTrackAsDefault="true"
        setFirstForcedSubtitleTrackAsDefault="true"
        insertFirstChapter00IfMissing="true"
    />

    <!-- Settings overridable in preferences -->
    <profileSettings
        app_DefaultSelectionString="-sel:all,+sel:(eng|favlang|nolang),-sel:(havemulti|havelossless|core|forced|lossy*(!eng)), -sel:subtitle*(!eng), -sel:mvcvideo,-sel:special, =100:all,-10:favlang"
    />

    <!-- Output formats currently supported by MakeMKV -->
    <outputSettings name="copy" outputFormat="directCopy">
        <description lang="eng">Copy track as is</description>
        <description lang="ger">Track 1:1 kopieren</description>
    </outputSettings>

    <outputSettings name="lpcm" outputFormat="LPCM-raw">
        <description lang="eng">Save as raw LPCM</description>
        <description lang="ger">Als RAW LPCM speichern</description>
    </outputSettings>

    <outputSettings name="wavex" outputFormat="LPCM-wavex">
        <description lang="eng">Save as LPCM in WAV container</description>
        <description lang="ger">Als LPCM im WAV-Container speichern</description>
    </outputSettings>

    <outputSettings name="flac-best" outputFormat="FLAC">
        <description lang="eng">Save as FLAC (best compression)</description>
        <description lang="ger">Als FLAC speichern (höchste Komprimierungsstufe)</description>
        <extraArgs>-compression_level 12</extraArgs>
    </outputSettings>

    <outputSettings name="flac-fast" outputFormat="FLAC">
        <description lang="eng">Save as FLAC (fast compression)</description>
        <extraArgs>-compression_level 5</extraArgs>
    </outputSettings>

    <!-- Default rule - copy as is -->
    <trackSettings input="default">
        <output outputSettingsName="copy"
                defaultSelection="$app_DefaultSelectionString">
        </output>
    </trackSettings>

    <!-- Save LPCM mono or stereo as raw LPCM -->
    <trackSettings input="LPCM-stereo">
        <output outputSettingsName="flac-best"
                defaultSelection="$app_DefaultSelectionString">
        </output>
    </trackSettings>

    <!-- Put multi-channel LPCM into WAVEX container-->
    <trackSettings input="LPCM-multi">
        <output outputSettingsName="flac-fast"
                defaultSelection="$app_DefaultSelectionString">
        </output>
    </trackSettings>

</profile>
I will say though, that I only convert PCM tracks to FLAC.
Since Media Center can decode DTS-HD and TrueHD anyway, it didn't seem worth the extra time to save ~1GB - though I will admit, I haven't really timed MakeMKV doing this.
 
There is additional downmix metadata which these formats store that Media Center may some day be capable of reading. (I think MakeMKV preserves this?)

In my entire library, it would probably free up the space for less than ten more films, so I'd rather just leave it in the original format for now. It wouldn't be difficult to convert at a later date.

[mwillems]

Thanks much for the advice (and the preset file).  I'll try playing around with handbrake and see if I can get better results

Any time I tried this though, I ended up turning the encoder settings up as high as they went, and after settling on a CRF setting which looked good, the file size was almost the same if not larger than the original.

That's been my experience so far too.

I will say though, that I only convert PCM tracks to FLAC.
Since Media Center can decode DTS-HD and TrueHD anyway, it didn't seem worth the extra time to save ~1GB - though I will admit, I haven't really timed MakeMKV doing this.
 
There is additional downmix metadata which these formats store that Media Center may some day be capable of reading. (I think MakeMKV preserves this?)

In my entire library, it would probably free up the space for less than ten more films, so I'd rather just leave it in the original format for now. It wouldn't be difficult to convert at a later date.

After processing about 50 files, my average savings from recompressing DTS-HD and True HD is about 8%, or about 2GB on average per film.  I did it as a separate step because I didn't know about the functionality at first.  Just doing recompression, makemkv takes about ten minutes per film to do the recompression, it doesn't seem to add quite that much when done as part of the initial processing.  The savings for LPCM is obviously significantly more.  In all I wound up saving 150GB of space recompressing (about 100 GB of that from DTS-MA and True HD sources) for a moderate time investment, which was worth it for me.

I didn't realize that I might be throwing away downmix data, though, which is kind of a drag.  

One bonus is that during reencoding MakeMKV caught a bunch of AV sync errors that weren't corrected in the original processing, and the movies now have tighter lipsync than they did before (I A/Bed at the affected intervals, and it was definitely better).

[glynor]

So here are some of the issues as I understand them (please correct me if any of these premisses are incorrect):  

1) Most commercial video is compressed using a lossy compression method

Correction (though largely irrelevant to the discussion):

All commercial video is compressed using a lossy compression method.  Often, multiple times (once on capture/editing, assuming a digital and not film-and-edl based workflow, and again for output).

[6233638]

After processing about 50 files, my average savings from recompressing DTS-HD and True HD is about 8%, or about 2GB on average per film.  I did it as a separate step because I didn't know about the functionality at first.  Just doing recompression, makemkv takes about ten minutes per film to do the recompression, it doesn't seem to add quite that much when done as part of the initial processing.  The savings for LPCM is obviously significantly more.  In all I wound up saving 150GB of space recompressing (about 100 GB of that from DTS-MA and True HD sources) for a moderate time investment, which was worth it for me.

That's more than I expected... and actually would be worthwhile.

I didn't realize that I might be throwing away downmix data, though, which is kind of a drag.

Well I don't know if it's preserved anyway, and Hendrik did not seem particularly interested in implementing the feature last time I bugged him about it.

All* commercial video is compressed using a lossy compression method.

*except RED footage

[glynor]

*except RED footage

Sure it is, for delivery.  The BluRay you get isn't raw red footage, nor is the hard drive or electronically delivered copy the movie theaters get.  Your digital cable is certainly heavily lossily compressed.  And, if you record it from broadcast TV, they're getting it via satellite or cable, and it is being sent to them compressed.

Unless you're counting stock footage as commercial (which I suppose it is, technically, but it will then be used to make a product that is lossy compressed), and even most of that isn't in formats that weren't ever compressed (even the nice copies).

There are plenty of other digital production examples of lossless compression (or completely uncompressed source footage).  Digital effects often requires it for rotoscoping and green screen and whatnot, for example.  That wasn't my point, though (and is why I said usually, but not always, twice).

There is no mainstream consumer video product, however, that doesn't have at least one round (and usually multiple, or a mixture) of lossy compression done to it before it gets to your eyes.  Unless you're making the video, you don't have lossless sources, and even then, it is almost certainly a giant jumble of different formats from different sources.

[mwillems]

That's more than I expected... and actually would be worthwhile.

It obviously varies a lot by movie.  Some I'll only get .5 GB, but with others I'll get 4 GB!  If I had done two or three and gotten less than 1GB, I might have given up the project, but the first two I did yielded better than 2GB, so I figured it'd be worth it to do all of them.

[6233638]

It obviously varies a lot by movie.  Some I'll only get .5 GB, but with others I'll get 4 GB!  If I had done two or three and gotten less than 1GB, I might have given up the project, but the first two I did yielded better than 2GB, so I figured it'd be worth it to do all of them.

Hmm, it looks like I can just run the files through MakeMKV again to convert the audio from DTS-HD/TrueHD to FLAC, so that might not be so bad.
 
I'll have to check and see if it preserves the specific flags set for my rips first.
 
P.S. I think that preset I created uses high compression FLAC for stereo, and "fast" compression for multichannel audio - though I'll probably be editing it now anyway.
 
The main thing is that it got the correct audio & subtitle streams for most discs automatically, and saved me having to uncheck a lot of options each time.

The BluRay you get isn't raw red footage, nor is the hard drive or electronically delivered copy the movie theaters get.

That's true. I only meant to say that RED footage is actually lossless as they store raw data. Obviously that never makes it out of the studio.
There are other cameras also claiming to shoot raw video, but it seemed more like they were storing 16-bit processed images, rather than truly raw footage. (i.e. before debayering and any other internal processes)

[jmone]

For me I just rip it all without any re encoding.  Disk is cheap.  WIting on my first 6tb hdd to arrive ($300) and say an average BD is 40gb this HDD will hold 150bd or $2ea.  Each 1gb you shave off is only saving 5cents

[6233638]

Well I just did a test conversion using the highest compression FLAC setting in MakeMKV.
It took about fifteen minutes. Normally converting an ISO to MKV takes about three.
 
While the savings will vary, in this case it only shaved off 700MB.
 
In future I might rip with this option enabled, but I won't be converting my existing library unless I really need the space.

[mwillems]

For me I just rip it all without any re encoding.  Disk is cheap.  WIting on my first 6tb hdd to arrive ($300) and say an average BD is 40gb this HDD will hold 150bd or $2ea.  Each 1gb you shave off is only saving 5cents

You're right, but that assumes an infinitely scaleable system and free backups.  I like to have a mirrored pool in another machine, which means I buy all my space twice, and without getting additional computers/hard drive controllers I have a limited number of ports available.

In this case I freed up 150 GB by clicking a few times between doing other things.  Granted, in your example that's only $7.50 worth of space, or with full redundancy, $15.  I probably wouldn't have expended a ton of effort for $15, but it was basically free and I was stuck sitting in front of the monitor anyway. My wife and I just had our first child this week, and she doesn't sleep right now unless she's being held.  My computer chair is one of the only rocking chairs in the house, so I had my daughter tucked up in my left arm rocking, and was mousing about generating GB with my free hand 

Well I just did a test conversion using the highest compression FLAC setting in MakeMKV.
It took about fifteen minutes. Normally converting an ISO to MKV takes about three.
 
While the savings will vary, in this case it only shaved off 700MB.
 
In future I might rip with this option enabled, but I won't be converting my existing library unless I really need the space.

There were definitely a few like that for me; it seems like different companies tended to do a better or worse job of compressing the audio, which seems weird.  For example, I consistently got multiple GB of savings with every part of a certain series, but less than a GB with each installment of a different series.  It may just be coincidence, but there seemed to be correlation.  If I had had several hundred to process, I might not have bothered either, but seventy seemed just enough to be achievable

[jmone]

You're right, but that assumes an infinitely scaleable system and free backups.  I like to have a mirrored pool in another machine, which means I buy all my space twice, and without getting additional computers/hard drive controllers I have a limited number of ports available.

In this case I freed up 150 GB by clicking a few times between doing other things.  Granted, in your example that's only $7.50 worth of space, or with full redundancy, $15.  I probably wouldn't have expended a ton of effort for $15, but it was basically free and I was stuck sitting in front of the monitor anyway. My wife and I just had our first child this week, and she doesn't sleep right now unless she's being held.  My computer chair is one of the only rocking chairs in the house, so I had my daughter tucked up in my left arm rocking, and was mousing about generating GB with my free hand 

...and of course this stuff is fun!  FYI I too have a second pool (both are currently 30+TB built using mostly 4TB .. some 3tb HDD) and I have pretty well maxed out my ports, but with the advent of 6TB HDD it will give a nice vol boost without any more infrastructure.   

[6233638]

There were definitely a few like that for me; it seems like different companies tended to do a better or worse job of compressing the audio, which seems weird.  For example, I consistently got multiple GB of savings with every part of a certain series, but less than a GB with each installment of a different series.  It may just be coincidence, but there seemed to be correlation.  If I had had several hundred to process, I might not have bothered either, but seventy seemed just enough to be achievable

Are you sure that the savings are from converting the audio to FLAC?
 
The biggest space savings I've found are from simply ripping out the main title and dropping the audio tracks you don't want. On some discs this can be as much as 30GB.

[mwillems]

Are you sure that the savings are from converting the audio to FLAC?
 
The biggest space savings I've found are from simply ripping out the main title and dropping the audio tracks you don't want. On some discs this can be as much as 30GB.

I'm 100% sure that the 150GB was from FLAC conversion, because  I did the FLAC conversion as a separate step on existing MKV files, because I didn't know FLAC conversion was an option at first.  I separated the main titles out a few weeks ago, and only realized later that MakeMKV could do the FLAC conversion, so I performed just the FLAC conversion on about 70 existing files that only had the main feature and the main audio track.  Only about 20 or so of those files were LPCM and those generated 50GB in savings; the remaining 50 mkvs generated a little less than 100 GB of the total.

I agree that the biggest space savings were definitely from separating the main title and discarding the audio tracks, but the FLAC conversion shaved off a good bit more (as noted about 8% or 2GB on average). 

[Hendrik]

With 20 out of 70 being PCM tracks, your percentage is really inflated quite a bit from that. Viewing audio-only, you can easily save over 50% on a PCM track, while the savings on TrueHD and DTS-HD tracks are significantly smaller.

A statistic on how much in total only the recompression of those saved might be more interesting, without the PCM files in the same count.
I doubt its very significant, maybe 10-20% over the audio, but the audio is only ~10% of a movie in size, so you do the math.

Also its maybe important to note that DTS-HD is typically bigger than TrueHD, so the potential gains are bigger as well.

[mwillems]

With 20 out of 70 being PCM tracks, your percentage is really inflated quite a bit from that. Viewing audio-only, you can easily save over 50% on a PCM track, while the savings on TrueHD and DTS-HD tracks are significantly smaller.

A statistic on how much in total only the recompression of those saved might be more interesting, without the PCM files in the same count.
I doubt its very significant, maybe 10-20% over the audio, but the audio is only ~10% of a movie in size, so you do the math.

Also its maybe important to note that DTS-HD is typically bigger than TrueHD, so the potential gains are bigger as well.

Sorry, I wasn't clear, I did separate out the non-PCM files and check the savings.  I kept all my original files so I could compare.  The 50 non-PCM files generated about 100GB of the total savings, so about 2GB a piece.  

I only got an average of 2.5 GB in average savings from the LPCM files, but I think that's because almost all of my LPCM movies are older and so have smaller file sizes to start with, and tend to be in mono or stereo.

The 8% figure was the combined average savings, the percent for just the non-PCM portion was closer to 6%.

[RussellS]

Re-encoding with high quality settings takes a while on my PC, so blind trial and error is inconvenient and frustrating.  I guess I'm just looking for some places to start that are more likely to produce good results than just blundering around.  Thanks in advance for any help.

The approach I took was to rip a Blu-ray using MakeMKV and then select a short 5 minute section of the movie with some dark scenes that will show up compression artifacts ets. Then use Handbrake to just re-encode this 5 minute section multiple times using different settings each time. This way the re-encoding only takes a couple of minutes each time and you can then do some direct A-B comparison tests on the same section of movie.

The problem is that is you ask ten people what their ideal compression settings are you will get ten different answers. It all depends on the size of the display used to view the movie, the resolution of the display, the viewing distance, your own personal tolerance to compression artifacts, disk space available and personal wealth to afford more disk space (cheap is relative). All these factors will influence what an acceptable compression ratio is and will normally be some sort of compromise between all those factors.

There was also one other factor that influenced my decision to compress and that was network bandwidth. I have two main rooms each with an MC system that plays media from the MC server located upstairs. In one of the rooms I do not have a wired network connection so I originally used WiFi and then, to improve bandwidth, I used a Homeplug mains wiring networking solution. However I found that neither of these options would reliably stream a Blu-ray in it's original ripped state (i.e. not re-encoded) and would occasionally stutter. Therefore I had to re-encode my Blu-Ray's not to save space but to save on required network bandwidth so they would stream reliably.

[mattkhan]

The problem is that is you ask ten people what their ideal compression settings are you will get ten different answers. It all depends on the size of the display used to view the movie, the resolution of the display, the viewing distance, your own personal tolerance to compression artifacts, disk space available and personal wealth to afford more disk space (cheap is relative). All these factors will influence what an acceptable compression ratio is and will normally be some sort of compromise between all those factors

this suggests that acceptable settings now may prove to be unacceptable after some future display upgrade doesn't it?

[6233638]

this suggests that acceptable settings now may prove to be unacceptable after some future display upgrade doesn't it?

That's exactly why I wouldn't recompress video.

[RussellS]

this suggests that acceptable settings now may prove to be unacceptable after some future display upgrade doesn't it?

I would totally agree with that statement.

That's exactly why I wouldn't recompress video.

In an ideal world neither would I. Unfortunately we don't live in an ideal world and sometimes circumstances dictate that compromises have to be made. For me personally disc space isn't an issue but network bandwidth is. We will be moving house next year and I will make sure that there are wired network connections in every room. When that happens I will re-rip my Blu-rays. For other people disc space may be the issue with limited funds to expand it. It's all very well saying that disc space is cheap but that is only the case if you have the income to be able to afford, and more importantly justify, spending money on extra disc space.
Therefore , for some people re-compressing is necessary and they need to find an acceptable compromise between space required and acceptable video quality (and in my case network bandwidth requirements).