Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[mvandyke]

I have a fairly large Music, Video library and have written tags to all of my files including Lyrics, Album Art and extensive AMG data (bio, reviews, etc.) pulled through MCutils.  Video files are in side car files of course.

The one thing that I have not done in the past is Analyze Audio.  I ran it through on all of my audio files and the DB size increased from 64MB to 222mb.  No new updates for files were added at this time only Analyze Audio and to write those tags to file.  I do see that the Waveform has a ton of text in it. Thousands of video files (Home Movies, Movies, etc.) were not updated during this process.

Would you expect this increase in the DB by 4X.  It's running OK but that is a huge jump and was surprised in just how large the increase was.

Thanks
Matt

[MGD_King]

I would expect the size to increase that much after analyzing your audio for the first time because the waveform information gets stored in your database.

[zybex]

The audio Analyze process adds the Waveform data to the database. This is used to display the full waveform when playing the song (right-click on the top Display banner -> Show Waveform).

MC seems to store 2 samples per second in that field, and each sample is 6 chars (a decimal number representing the average height of the bar at that point in time); the waveform.jmd database is compressed at around 40% rate (YMMV). So for your DB to grow by 160MB, it means you likely have around 160e6/6/2/3600/0.4 = 9200 hours of music. Lets say between 8500h to 9500h... am I close?

[mvandyke]

JRiver says about 362 days so quite a bit.

Thanks for the quick response from the team!

Matt

[zybex]

362 days = 8688 hours! Yaaay for math 

[zybex]

Here's a question for JRiver (or someone who knows): is this waveform shown/used anywhere else? Is it ever shown with higher resolution than that less-than-24-pixel height bar?

Because if not... this waveform could be stored in a much more compact form. I just tested an encoding to store the same info using 1/6th the DB size (3MB instead of 18MB for 1000 hours). Basically, instead of storing floats it would store a single char per sample representing a value between 0 and 95, which would correspond to the height of the bar in pixels for that sample. Since the waveform is symmetrical this allows for a max height of up to 190 pixels without any loss in detail relative to the current solution.

I tested this in Excel - screenshot attached; Blue is the original MC data, Green is the same with Integers, Red is the Char representation. The waveforms look exactly the same.

Here's a snippet of an MC waveform - 192 bytes:

Code: [Select]

.1216;.1196;.0801;.1209;.1271;.0963;.1181;.0964;.084;.1087;.1182;.1221;.1475;.094;.0389;.1233;.1798;.084;.1539;.1601;.0982;.182;.1137;.2474;.2649;.2555;.2385;.2192;.2216;.244;.2225;.2318;.2161
Here's the equivalent data using integers only (height of the bar in pixels) - 98 bytes:

Code: [Select]

36;36;24;36;38;29;35;29;25;33;35;37;44;28;12;37;54;25;46;48;29;55;34;74;79;77;72;66;66;73;67;70;65
And the same data encoded using 1 char per number (from #32 to #127) - 33 bytes only:

Code: [Select]

DD8DF=C=9ACEL<,EV9NP=WBjomhbbicfa
The Waveform field is not meant to be edited directly by users so it doesn't need to be readable.
For compatibility, MC could add a prefix like "WAV2:" to that string. This would allow it to parse both old and new Waveforms.

[Hendrik]

We have some plans to improve the compression of the waveform storage, although I don't want to reduce the precision based on todays UI details (eg. we'll continue storing a value based on the audio, not the height its being rendered at), but we can certainly reduce the space a lot by not storing number strings.

[DJLegba]

It wouldn't increase my database size if there was an option to turn it off.

[Hendrik]

Audio Analysis is entirely optional.

[wer]

DJ is talking about saving waveform data. Audio Analysis isn't really optional if you want volume leveling.

[DJLegba]

Not only saving waveform data, but doing the analysis in the first place. It's done to support an optional visual display that I don't use.

[wer]

Audio Analysis is done to support Volume Leveling.  Waveform is new gravy.  Audio Analysis is required for Volume Leveling.

If you don't ever want Volume Leveling, you can indeed turn Audio Analysis totally off, as Hendrik said.

You will then lose the Waveform, the Volume Leveling, and the storage hit.

[DJLegba]

Audio Analysis is done to support Volume Leveling.  Waveform is new gravy.  Audio Analysis is required for Volume Leveling.

If you don't ever want Volume Leveling, you can indeed turn Audio Analysis totally off, as Hendrik said.

You will then lose the Waveform, the Volume Leveling, and the storage hit.

Correct. I meant doing the waveform analysis. Don't need it, don't want it. Audio analysis is fine.

[EnglishTiger]

Whilst I agree that Audio Analysis is Important, even essential, adding in the "Compulsory" Waveform tag/field data was a badly thought out reaction to a request by a minority of users.

As Zybex said, it contains 2 values per second, which means for a track that is 20 minutes in length there will be 1,200 pairs of values. On my 26" screen where the Progress Bar is 380mm long that equates to just over 3 samples per mm. Which makes is about as useful as the proverbial "extra hole in the head"; the spectrum analyser shows far more meaningful information than that silly waveform ever will.

Oh and mvandyke, it's probably not just the database that went up in size, the sizes of all the files you analysed may have gone up as well; because the MC default is to write the results of the Waveform analysis to the Database and the Tags(Files).

Creating the waveform has been modified twice already, once during "beta" testing to allow users to change it's colours, and once to "reduce it's size" after it was discovered it was causing extended track/library load times especially with files on external discs. Now we find out it's going to be modified yet again to reduce it's size even more. Yet another example of the MC Development Team creating more work for themselves by not working out all the probable consequences of their actions before picking up a coding pen.

In addition if you check through the release history for MC 26 you will spot that the MC Team started work on reducing the size of "; delimited list" tags/fields by removing the space between entries after they had imposed that "extremely large" waveform field/tag on us. Yet another thing that caused more work because they ended up, as a result of a lot of users complaining, because they had to go through the system/code adding that space back in everywhere those fields/tags appear on screen, work that still hasn't been completed. Oh, MC craftiness with the waveform field/tag didn't end there, in the database it's Edit Type is set to "Standard" so in both the tag window and any view which exposes that tag/field the user only ever gets to see a "small sample" of the data it holds.

What needs doing, a lot sooner than the planned modifications to the waveform size, is to make Creating the Waveform field/tag OPTIONAL in the same way that Analyze for HDCD is. So that those of us who don't want that pretty (ugly) alternative to the progress bar in the Player Window don't have to suffer having to remove data that we don't need or want from the database and files.

[Awesome Donkey]

You can mostly workaround it by going to MC's Options > Library & Folders > Manage Library Fields and selecting Waveform in the list, and unchecking Save in file tags (if possible). This will prevent MC from writing the Waveform tags in the files themselves, but it'll still be saved in the database instead. From there you can select all files, right click and select Tag, and in the tag list scroll down to Waveform and right click on the field and select Empty to remove the Waveform tags.

Of course, if your files already contain the Waveform tags written to them and you want to remove all traces, you'll want to empty the Waveform tag data before changing the Waveform library field setting to prevent writes of the Waveform tag to files.

Every few days I clear out the Waveform data added to the database from newly imported files, and since it's not being written to the file's metadata it makes cleaning up Waveform tags from the database pretty straightforward.

[zybex]

We have some plans to improve the compression of the waveform storage, although I don't want to reduce the precision based on todays UI details (eg. we'll continue storing a value based on the audio, not the height its being rendered at), but we can certainly reduce the space a lot by not storing number strings.

Hendrik, thanks for taking the time to respond.
If there are plans to use the extra precision for some other feature, that's good. But if it's just because you might need it in the future for a larger display... you could store it now with just 6/7/8 bit precision (for the current needs), and if the need arises for more in a future version, the users can always rerun the analysis to regenerate with higher precision.

[lepa]

Every few days I clear out the Waveform data added to the database from newly imported files, and since it's not being written to the file's metadata it makes cleaning up Waveform tags from the database pretty straightforward.

I do this also but it would be nice if didn't have to...

[zybex]

From there you can select all files, right click and select Tag, and in the tag list scroll down to Waveform and right click on the field and select Empty to remove the Waveform tags.

An alternative way to clear it is to close MC and delete the "Field (waveform).jmd" file. MC doesn't seem to complain. So you can just schedule a task in Windows to [try to] delete the file every day or at Windows startup

[zybex]

I agree with some of EnglishTiger's criticism, but it's not so simple. Software development is an iterative process and there will always be unforeseen bugs/issues/problems.

JRiver is [usually] very responsive in implementing requested features and bug fixes - if you compare it with the industry norm of having 1 or 2 releases per year... it's clear they chose responsiveness instead, what some call an AGILE process. This of course comes at a cost of less testing, and less planning/design. It's a trade-off.

I do think that some features should have more planning behind them, and there should be a better strategy or roadmap for feature development, and even a planned rewrite of some modules (and maybe there is already; they don't need to tell users everything). Most large codebases evolve organically over time until they become quite messy, and a rewrite is often the only way to cleanup, speedup, and add space for new features. Keep in mind though, that any rewrite also usually implies some loss of functionality - parity with the old codebase takes time to achieve. It's an expensive process, so it's a big/controversial decision to make.

Where I think JRiver could definitely improve is on the communication department, particularly in how they handle feature requests. The current approach is more about ignoring posts, where sometimes a simple reply saying "thanks, but no thanks" or "we'll consider it" would suffice. This ACK is all that is missing, in my view. As a suggestion, what about creating a top-level Forum called "Feature Requests", where each post is a request to be answered Yay or Nay? Sort of like that recent thread where you asked for "small things we could've done".

[Manfred]

I love the waveform! A few GB more storage on SSD or waiting a second that the lib loads to a MC client - It really doesn't matter for me.

[dtc]

Analyze Audio used to just do the volume level/R128 calculations. But HDCD analysis and waveform analysis have been added. Unfortunately, these additional features are not options. If you want to do volume leveling you also have to do waveform analysis.  A couple of check boxes and a few if statements in the code would make Analyze Audio much more user friendly.

[Awesome Donkey]

An alternative way to clear it is to close MC and delete the "Field (waveform).jmd" file. MC doesn't seem to complain. So you can just schedule a task in Windows to [try to] delete the file every day or at Windows startup

Does setting the file to be read-only not prevent it as well?

[zybex]

Didn't test that. That may cause MC to crash or have an inconsistent internal state due to failed writes.

[Awesome Donkey]

Tested it, it doesn't work. MC must automatically be changing the attribute of the file back from read-only.

[mattkhan]

I agree it should be an option but even if it's not, generating and storing this data for video files seems completely pointless to me as you'll never see it.

[comox]

I just finished re-analyzing my entire library.

If I had kept the waveform data the size of my library would have increased 26x from 30MB to 800MB.

This size increase is unacceptable to me because I often backup my library several times a day.

Deleting "field (waveform).jmd" is a good work around.