INTERACT FORUM

Please login or register.

Login with username, password and session length
Advanced search  
Pages: [1]   Go Down

Author Topic: Analyze Audio function increased JR DataBase size by 4x  (Read 2123 times)

mvandyke

  • World Citizen
  • ***
  • Posts: 159
Analyze Audio function increased JR DataBase size by 4x
« on: January 13, 2021, 10:18:24 am »

I have a fairly large Music, Video library and have written tags to all of my files including Lyrics, Album Art and extensive AMG data (bio, reviews, etc.) pulled through MCutils.  Video files are in side car files of course.

The one thing that I have not done in the past is Analyze Audio.  I ran it through on all of my audio files and the DB size increased from 64MB to 222mb.  No new updates for files were added at this time only Analyze Audio and to write those tags to file.  I do see that the Waveform has a ton of text in it. Thousands of video files (Home Movies, Movies, etc.) were not updated during this process.

Would you expect this increase in the DB by 4X.  It's running OK but that is a huge jump and was surprised in just how large the increase was.

Thanks
Matt

Logged

MGD_King

  • Regular Member
  • Citizen of the Universe
  • *****
  • Posts: 548
  • It's not easy being me, but it sure is fun!
Re: Analyze Audio function increased JR DataBase size by 4x
« Reply #1 on: January 13, 2021, 10:55:16 am »

I would expect the size to increase that much after analyzing your audio for the first time because the waveform information gets stored in your database.
Logged

zybex

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 2619
Re: Analyze Audio function increased JR DataBase size by 4x
« Reply #2 on: January 13, 2021, 10:57:10 am »

The audio Analyze process adds the Waveform data to the database. This is used to display the full waveform when playing the song (right-click on the top Display banner -> Show Waveform).

MC seems to store 2 samples per second in that field, and each sample is 6 chars (a decimal number representing the average height of the bar at that point in time); the waveform.jmd database is compressed at around 40% rate (YMMV). So for your DB to grow by 160MB, it means you likely have around 160e6/6/2/3600/0.4 = 9200 hours of music. Lets say between 8500h to 9500h... am I close? :)
Logged

mvandyke

  • World Citizen
  • ***
  • Posts: 159
Re: Analyze Audio function increased JR DataBase size by 4x
« Reply #3 on: January 13, 2021, 11:24:16 am »

JRiver says about 362 days so quite a bit.

Thanks for the quick response from the team!

Matt
Logged

zybex

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 2619
Re: Analyze Audio function increased JR DataBase size by 4x
« Reply #4 on: January 13, 2021, 11:29:06 am »

362 days = 8688 hours! Yaaay for math  8)
Logged

zybex

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 2619
Re: Analyze Audio function increased JR DataBase size by 4x
« Reply #5 on: January 13, 2021, 12:09:01 pm »

Here's a question for JRiver (or someone who knows): is this waveform shown/used anywhere else? Is it ever shown with higher resolution than that less-than-24-pixel height bar?

Because if not... this waveform could be stored in a much more compact form. I just tested an encoding to store the same info using 1/6th the DB size (3MB instead of 18MB for 1000 hours). Basically, instead of storing floats it would store a single char per sample representing a value between 0 and 95, which would correspond to the height of the bar in pixels for that sample. Since the waveform is symmetrical this allows for a max height of up to 190 pixels without any loss in detail relative to the current solution.

I tested this in Excel - screenshot attached; Blue is the original MC data, Green is the same with Integers, Red is the Char representation. The waveforms look exactly the same.

Here's a snippet of an MC waveform - 192 bytes:
Code: [Select]
.1216;.1196;.0801;.1209;.1271;.0963;.1181;.0964;.084;.1087;.1182;.1221;.1475;.094;.0389;.1233;.1798;.084;.1539;.1601;.0982;.182;.1137;.2474;.2649;.2555;.2385;.2192;.2216;.244;.2225;.2318;.2161
Here's the equivalent data using integers only (height of the bar in pixels) - 98 bytes:
Code: [Select]
36;36;24;36;38;29;35;29;25;33;35;37;44;28;12;37;54;25;46;48;29;55;34;74;79;77;72;66;66;73;67;70;65
And the same data encoded using 1 char per number (from #32 to #127) - 33 bytes only:
Code: [Select]
DD8DF=C=9ACEL<,EV9NP=WBjomhbbicfa
The Waveform field is not meant to be edited directly by users so it doesn't need to be readable.
For compatibility, MC could add a prefix like "WAV2:" to that string. This would allow it to parse both old and new Waveforms.
Logged

Hendrik

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 10939
Re: Analyze Audio function increased JR DataBase size by 4x
« Reply #6 on: January 13, 2021, 06:43:14 pm »

We have some plans to improve the compression of the waveform storage, although I don't want to reduce the precision based on todays UI details (eg. we'll continue storing a value based on the audio, not the height its being rendered at), but we can certainly reduce the space a lot by not storing number strings.
Logged
~ nevcairiel
~ Author of LAV Filters

DJLegba

  • Citizen of the Universe
  • *****
  • Posts: 995
Re: Analyze Audio function increased JR DataBase size by 4x
« Reply #7 on: January 13, 2021, 07:00:32 pm »

It wouldn't increase my database size if there was an option to turn it off.
Logged

Hendrik

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 10939
Re: Analyze Audio function increased JR DataBase size by 4x
« Reply #8 on: January 13, 2021, 07:56:42 pm »

Audio Analysis is entirely optional.
Logged
~ nevcairiel
~ Author of LAV Filters

wer

  • Citizen of the Universe
  • *****
  • Posts: 2640
Re: Analyze Audio function increased JR DataBase size by 4x
« Reply #9 on: January 13, 2021, 08:06:05 pm »

DJ is talking about saving waveform data. Audio Analysis isn't really optional if you want volume leveling.
Logged

DJLegba

  • Citizen of the Universe
  • *****
  • Posts: 995
Re: Analyze Audio function increased JR DataBase size by 4x
« Reply #10 on: January 13, 2021, 10:06:46 pm »

Not only saving waveform data, but doing the analysis in the first place. It's done to support an optional visual display that I don't use.
Logged

wer

  • Citizen of the Universe
  • *****
  • Posts: 2640
Re: Analyze Audio function increased JR DataBase size by 4x
« Reply #11 on: January 13, 2021, 10:32:55 pm »

Audio Analysis is done to support Volume Leveling.  Waveform is new gravy.  Audio Analysis is required for Volume Leveling.

If you don't ever want Volume Leveling, you can indeed turn Audio Analysis totally off, as Hendrik said.

You will then lose the Waveform, the Volume Leveling, and the storage hit.
Logged

DJLegba

  • Citizen of the Universe
  • *****
  • Posts: 995
Re: Analyze Audio function increased JR DataBase size by 4x
« Reply #12 on: January 13, 2021, 10:48:08 pm »

Audio Analysis is done to support Volume Leveling.  Waveform is new gravy.  Audio Analysis is required for Volume Leveling.

If you don't ever want Volume Leveling, you can indeed turn Audio Analysis totally off, as Hendrik said.

You will then lose the Waveform, the Volume Leveling, and the storage hit.

Correct. I meant doing the waveform analysis. Don't need it, don't want it. Audio analysis is fine.
Logged

EnglishTiger

  • Regular Member
  • Citizen of the Universe
  • *****
  • Posts: 1088
Re: Analyze Audio function increased JR DataBase size by 4x
« Reply #13 on: January 14, 2021, 12:33:44 am »

Whilst I agree that Audio Analysis is Important, even essential, adding in the "Compulsory" Waveform tag/field data was a badly thought out reaction to a request by a minority of users.

As Zybex said, it contains 2 values per second, which means for a track that is 20 minutes in length there will be 1,200 pairs of values. On my 26" screen where the Progress Bar is 380mm long that equates to just over 3 samples per mm. Which makes is about as useful as the proverbial "extra hole in the head"; the spectrum analyser shows far more meaningful information than that silly waveform ever will.

Oh and mvandyke, it's probably not just the database that went up in size, the sizes of all the files you analysed may have gone up as well; because the MC default is to write the results of the Waveform analysis to the Database and the Tags(Files).

Creating the waveform has been modified twice already, once during "beta" testing to allow users to change it's colours, and once to "reduce it's size" after it was discovered it was causing extended track/library load times especially with files on external discs. Now we find out it's going to be modified yet again to reduce it's size even more. Yet another example of the MC Development Team creating more work for themselves by not working out all the probable consequences of their actions before picking up a coding pen.

In addition if you check through the release history for MC 26 you will spot that the MC Team started work on reducing the size of "; delimited list" tags/fields by removing the space between entries after they had imposed that "extremely large" waveform field/tag on us. Yet another thing that caused more work because they ended up, as a result of a lot of users complaining, because they had to go through the system/code adding that space back in everywhere those fields/tags appear on screen, work that still hasn't been completed. Oh, MC craftiness with the waveform field/tag didn't end there, in the database it's Edit Type is set to "Standard" so in both the tag window and any view which exposes that tag/field the user only ever gets to see a "small sample" of the data it holds.

What needs doing, a lot sooner than the planned modifications to the waveform size, is to make Creating the Waveform field/tag OPTIONAL in the same way that Analyze for HDCD is. So that those of us who don't want that pretty (ugly) alternative to the progress bar in the Player Window don't have to suffer having to remove data that we don't need or want from the database and files.
Logged
Apple Mac Mini Desktop Computer with M4 Pro chip with 12 core CPU and 16 core GPU: 24GB Unified Memory, 512GB SSD Storage, Gigabit Ethernet, 3 Thunderbolt5 + 2USBC ports.

Awesome Donkey

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 7810
  • Autumn shade...
Re: Analyze Audio function increased JR DataBase size by 4x
« Reply #14 on: January 14, 2021, 04:08:21 am »

You can mostly workaround it by going to MC's Options > Library & Folders > Manage Library Fields and selecting Waveform in the list, and unchecking Save in file tags (if possible). This will prevent MC from writing the Waveform tags in the files themselves, but it'll still be saved in the database instead. From there you can select all files, right click and select Tag, and in the tag list scroll down to Waveform and right click on the field and select Empty to remove the Waveform tags.

Of course, if your files already contain the Waveform tags written to them and you want to remove all traces, you'll want to empty the Waveform tag data before changing the Waveform library field setting to prevent writes of the Waveform tag to files.

Every few days I clear out the Waveform data added to the database from newly imported files, and since it's not being written to the file's metadata it makes cleaning up Waveform tags from the database pretty straightforward.
Logged
I don't work for JRiver... I help keep the forums safe from "male enhancements" and other sources of sketchy pharmaceuticals.

Windows 11 24H2 Update 64-bit + Ubuntu 24.10 Oracular Oriole 64-bit | Windows 11 24H2 Update 64-bit (Intel N305 Fanless NUC 16GB RAM/500GB M.2 NVMe SSD)
JRiver Media Center 33 (Windows + Linux) | iFi ZEN DAC 3 | JBL 306P MkII Studio Monitors | Audio-Technica ATH-M50x Headphones

zybex

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 2619
Re: Analyze Audio function increased JR DataBase size by 4x
« Reply #15 on: January 14, 2021, 05:20:11 am »

We have some plans to improve the compression of the waveform storage, although I don't want to reduce the precision based on todays UI details (eg. we'll continue storing a value based on the audio, not the height its being rendered at), but we can certainly reduce the space a lot by not storing number strings.

Hendrik, thanks for taking the time to respond.
If there are plans to use the extra precision for some other feature, that's good. But if it's just because you might need it in the future for a larger display... you could store it now with just 6/7/8 bit precision (for the current needs), and if the need arises for more in a future version, the users can always rerun the analysis to regenerate with higher precision.
Logged

lepa

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 2033
Re: Analyze Audio function increased JR DataBase size by 4x
« Reply #16 on: January 14, 2021, 05:22:29 am »

Every few days I clear out the Waveform data added to the database from newly imported files, and since it's not being written to the file's metadata it makes cleaning up Waveform tags from the database pretty straightforward.
I do this also but it would be nice if didn't have to...
Logged

zybex

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 2619
Re: Analyze Audio function increased JR DataBase size by 4x
« Reply #17 on: January 14, 2021, 05:24:01 am »

From there you can select all files, right click and select Tag, and in the tag list scroll down to Waveform and right click on the field and select Empty to remove the Waveform tags.

An alternative way to clear it is to close MC and delete the "Field (waveform).jmd" file. MC doesn't seem to complain. So you can just schedule a task in Windows to [try to] delete the file every day or at Windows startup ;)
Logged

zybex

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 2619
Re: Analyze Audio function increased JR DataBase size by 4x
« Reply #18 on: January 14, 2021, 05:43:09 am »

I agree with some of EnglishTiger's criticism, but it's not so simple. Software development is an iterative process and there will always be unforeseen bugs/issues/problems.

JRiver is [usually] very responsive in implementing requested features and bug fixes - if you compare it with the industry norm of having 1 or 2 releases per year... it's clear they chose responsiveness instead, what some call an AGILE process. This of course comes at a cost of less testing, and less planning/design. It's a trade-off.

I do think that some features should have more planning behind them, and there should be a better strategy or roadmap for feature development, and even a planned rewrite of some modules (and maybe there is already; they don't need to tell users everything). Most large codebases evolve organically over time until they become quite messy, and a rewrite is often the only way to cleanup, speedup, and add space for new features. Keep in mind though, that any rewrite also usually implies some loss of functionality - parity with the old codebase takes time to achieve. It's an expensive process, so it's a big/controversial decision to make.

Where I think JRiver could definitely improve is on the communication department, particularly in how they handle feature requests. The current approach is more about ignoring posts, where sometimes a simple reply saying "thanks, but no thanks" or "we'll consider it" would suffice. This ACK is all that is missing, in my view. As a suggestion, what about creating a top-level Forum called "Feature Requests", where each post is a request to be answered Yay or Nay? Sort of like that recent thread where you asked for "small things we could've done".
Logged

Manfred

  • Citizen of the Universe
  • *****
  • Posts: 1038
Re: Analyze Audio function increased JR DataBase size by 4x
« Reply #19 on: January 14, 2021, 06:14:31 am »

I love the waveform! A few GB more storage on SSD or waiting a second that the lib loads to a MC client - It really doesn't matter for me.
Logged
WS (AMD Ryzen 7 5700G, 32 GB DDR4-3200, 8=2x2+4 TB SDD, LG 34UC98-W)-USB|ADI-2 DAC FS|Canton AM5 - File Server (i3-3.9 GHz, 16GB ECC DDR4-2400, 46 TB disk space) - Media Renderer (i3-3.8 GHz, 8GB DDR4-2133, GTX 960)-USB|Devialet D220 Pro|Audeze LCD 2|B&W 804S|LG 4K OLED )

dtc

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 3119
Re: Analyze Audio function increased JR DataBase size by 4x
« Reply #20 on: January 14, 2021, 06:51:02 am »

Analyze Audio used to just do the volume level/R128 calculations. But HDCD analysis and waveform analysis have been added. Unfortunately, these additional features are not options. If you want to do volume leveling you also have to do waveform analysis.  A couple of check boxes and a few if statements in the code would make Analyze Audio much more user friendly.
Logged

Awesome Donkey

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 7810
  • Autumn shade...
Re: Analyze Audio function increased JR DataBase size by 4x
« Reply #21 on: January 14, 2021, 06:58:28 am »

An alternative way to clear it is to close MC and delete the "Field (waveform).jmd" file. MC doesn't seem to complain. So you can just schedule a task in Windows to [try to] delete the file every day or at Windows startup ;)

Does setting the file to be read-only not prevent it as well?
Logged
I don't work for JRiver... I help keep the forums safe from "male enhancements" and other sources of sketchy pharmaceuticals.

Windows 11 24H2 Update 64-bit + Ubuntu 24.10 Oracular Oriole 64-bit | Windows 11 24H2 Update 64-bit (Intel N305 Fanless NUC 16GB RAM/500GB M.2 NVMe SSD)
JRiver Media Center 33 (Windows + Linux) | iFi ZEN DAC 3 | JBL 306P MkII Studio Monitors | Audio-Technica ATH-M50x Headphones

zybex

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 2619
Re: Analyze Audio function increased JR DataBase size by 4x
« Reply #22 on: January 14, 2021, 07:04:16 am »

Didn't test that. That may cause MC to crash or have an inconsistent internal state due to failed writes.
Logged

Awesome Donkey

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 7810
  • Autumn shade...
Re: Analyze Audio function increased JR DataBase size by 4x
« Reply #23 on: January 14, 2021, 07:56:14 am »

Tested it, it doesn't work. MC must automatically be changing the attribute of the file back from read-only.
Logged
I don't work for JRiver... I help keep the forums safe from "male enhancements" and other sources of sketchy pharmaceuticals.

Windows 11 24H2 Update 64-bit + Ubuntu 24.10 Oracular Oriole 64-bit | Windows 11 24H2 Update 64-bit (Intel N305 Fanless NUC 16GB RAM/500GB M.2 NVMe SSD)
JRiver Media Center 33 (Windows + Linux) | iFi ZEN DAC 3 | JBL 306P MkII Studio Monitors | Audio-Technica ATH-M50x Headphones

mattkhan

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 4231
Re: Analyze Audio function increased JR DataBase size by 4x
« Reply #24 on: January 24, 2021, 04:47:39 am »

I agree it should be an option but even if it's not, generating and storing this data for video files seems completely pointless to me as you'll never see it.
Logged

comox

  • Galactic Citizen
  • ****
  • Posts: 428
Re: Analyze Audio function increased JR DataBase size by 4x
« Reply #25 on: January 24, 2021, 10:26:28 am »

I just finished re-analyzing my entire library.

If I had kept the waveform data the size of my library would have increased 26x from 30MB to 800MB.

This size increase is unacceptable to me because I often backup my library several times a day.

Deleting "field (waveform).jmd" is a good work around.
Logged
Pages: [1]   Go Up