INTERACT FORUM

Please login or register.

Login with username, password and session length
Advanced search  
Pages: [1]   Go Down

Author Topic: Database Optimizations: 12.0.492  (Read 4151 times)

Matt

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 42388
  • Shoes gone again!
Database Optimizations: 12.0.492
« on: May 02, 2008, 05:29:27 pm »

From the build 492 thread:

The filename field now takes advantage of the fact that other tags often appear in the filename.

As a real world example, previously AoXoMoXoA's 150,000 track library took around 21 MB in memory for the basic file data.  Over 18 MB of this was from the filename field.  With this new system, the filename field takes 800 KB, so the entire library takes around 3 MB.  That's a big difference.
Logged
Matt Ashland, JRiver Media Center

Matt

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 42388
  • Shoes gone again!
Database Optimizations: 12.0.492
« Reply #1 on: May 02, 2008, 06:52:51 pm »

Here's another data point from my home library for the database changes with this build:

Build 491: Database: CDataHolder::Load: Field: Filename; Files: 11498; Data bytes: 864256; Elapsed ms: 2.387
Build 492: Database: CDataHolder::Load: Field: Filename; Files: 11495; Data bytes: 36864; Elapsed ms: 1.070

Database memory usage wasn't a real issue before with only 11,xxx files, but the change causes 24x less memory for the field.  The filename field was over 50% of the total memory usage before, so the overall effect is sizable.
Logged
Matt Ashland, JRiver Media Center

Link

  • Regular Member
  • World Citizen
  • ***
  • Posts: 143
  • Is that some kind of Eastern thing?
Re: Database Optimizations: 12.0.492
« Reply #2 on: May 05, 2008, 12:50:43 pm »

From the build 492 thread:

The filename field now takes advantage of the fact that other tags often appear in the filename.

Does this mean there may be an "optimal" way of naming our files to reduce the memory impact?
Logged

Matt

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 42388
  • Shoes gone again!
Re: Database Optimizations: 12.0.492
« Reply #3 on: May 05, 2008, 01:12:12 pm »

Does this mean there may be an "optimal" way of naming our files to reduce the memory impact?

Memory usage will never get worse than before and will normally get much better, so there's no reason to worry too much.

With that said, keeping tag information in-step with the filenames will help.  Using a consistent naming approach also helps.

All my music is stored like:
M:\Music\APE\[Artist]\[Album]\[Artist] - [Album] - [track #] - [Name].ape

A shortened version of this string is stored one time on disk, and one time in memory while running.  It's used for _all_ the music in my database.

Before this change, the full-off expanded filename was stored for every file.
Logged
Matt Ashland, JRiver Media Center

jack wallstreet

  • Citizen of the Universe
  • *****
  • Posts: 522
Re: Database Optimizations: 12.0.492
« Reply #4 on: May 05, 2008, 01:42:33 pm »

This change looks very impressive.  Thank you!
Logged
John

AoXoMoXoA

  • Regular Member
  • Citizen of the Universe
  • *****
  • Posts: 1551
  • I am a kangaroo . . . . no, really!
Re: Database Optimizations: 12.0.492
« Reply #5 on: May 05, 2008, 01:44:47 pm »

So the file & folder naming structure, for optimum performance with a larger library, the shorter the better then?
Logged
. . . the game is rigged

Doof

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 5908
  • Farm Animal Stupid
Re: Database Optimizations: 12.0.492
« Reply #6 on: May 05, 2008, 01:47:13 pm »

Would it be possible for MC to insure that our filenames are in step with our tags? It would be really helpful to specify once that I want [Album Artist (auto)]\[Album]\[Artist] - [Name] and then have MC keep things synced if and when I change one of those tags.
Logged

Matt

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 42388
  • Shoes gone again!
Re: Database Optimizations: 12.0.492
« Reply #7 on: May 05, 2008, 02:57:45 pm »

So the file & folder naming structure, for optimum performance with a larger library, the shorter the better then?

Not really, since if you use the same long string over and over, it only gets stored once.
Logged
Matt Ashland, JRiver Media Center

Matt

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 42388
  • Shoes gone again!
Re: Database Optimizations: 12.0.492
« Reply #8 on: May 05, 2008, 02:59:48 pm »

Would it be possible for MC to insure that our filenames are in step with our tags? It would be really helpful to specify once that I want [Album Artist (auto)]\[Album]\[Artist] - [Name] and then have MC keep things synced if and when I change one of those tags.

I'd like this feature.

It's just that we're reticent to add anything that would automatically change filenames.  That's why it's an explicit user action now.  (and only really a few clicks to have it check / cleanup the entire library)
Logged
Matt Ashland, JRiver Media Center

hit_ny

  • Citizen of the Universe
  • *****
  • Posts: 3310
  • nothing more to say...
Re: Database Optimizations: 12.0.492
« Reply #9 on: May 05, 2008, 03:04:50 pm »

Does anyone notice any speedups after this change, if so where ?
Logged

AoXoMoXoA

  • Regular Member
  • Citizen of the Universe
  • *****
  • Posts: 1551
  • I am a kangaroo . . . . no, really!
Re: Database Optimizations: 12.0.492
« Reply #10 on: May 05, 2008, 03:16:24 pm »

Does anyone notice any speedups after this change, if so where ?
I believe I see an increase in browsing.
I have not really seen anything noticable in tagging.

It's just that we're reticent to add anything that would automatically change filenames.  That's why it's an explicit user action now.  (and only really a few clicks to have it check / cleanup the entire library)
Thank you!
I, for one, am very particular about not having anything automatically doing much of anything to my files without my explicitly initiating it.
Logged
. . . the game is rigged

newsposter

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 789
Re: Database Optimizations: 12.0.492
« Reply #11 on: May 05, 2008, 03:42:25 pm »

I'm on the other side of the filename issue.

I don't care a whit about the 'real world' name of a file.  It's the metadata describing the file contents that is important.

What would be nice if JR could come up with an 'optimized' auto-name/rename scheme for files.  You could even stick with a hashed filename to be able to stick with an 8+3 convention.

Embed the metadata wherever.  A filename 'key' could be internal to the actual file as well as being kept in the LS database **and** represented in the actual filename.  Think of it as a brute-force auxillary indexing method.  This scheme would (in theory anyway) improve database size and performance as well as recoverability from a blown database or NTFS directory structure.

We do this all the time with Oracle and DB2 databases behind some really huge SAP systems.  On those beasts it does improve performace, acts as an additional sanity check on dB consistency, and on more than one occasion has been a life saver recovering from some serious system (ok, people) errors.  Gotta love those junior DBAs........

When syncing to an external device/player, a 'sensible' filename could be restored/reconstructed from the metadata.  This is necessary because most/all players use the filename as a part of the user interface & presentation.  Downside, it adds an extra internal step to sync activities.  Upside, it auto-builds consistent filenames on a player.  It's not too far fetched to allow the setting of a few filters so that the auto-build of the new filenames could follow a user-defined pattern.

To keep everyone happy, the use of a scheme like this it would be an optional setting.
Logged

darichman

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 1362
Re: Database Optimizations: 12.0.492
« Reply #12 on: May 05, 2008, 08:14:55 pm »

I'd like this feature.

It's just that we're reticent to add anything that would automatically change filenames.  That's why it's an explicit user action now.  (and only really a few clicks to have it check / cleanup the entire library)

What if we could type expressions directly into the [Filename] field and MC could remember the expression? That way it would be a user choice at least and MC could remember how to rename files on a file by file basis.
Just a thought...
Logged

jack wallstreet

  • Citizen of the Universe
  • *****
  • Posts: 522
Re: Database Optimizations: 12.0.492
« Reply #13 on: May 05, 2008, 10:35:46 pm »


It's just that we're reticent to add anything that would automatically change filenames.  That's why it's an explicit user action now.  (and only really a few clicks to have it check / cleanup the entire library)


I concur with Matt.  I do a lot of frequent tag changes and digital tweaks to files where I overwrite as well as constantly add files.  I hope J River remains "reticent" to add anything that would automatically change filenames..  There would possibly be some advantage to me that would make the work I do more efficient, but I think the danger would outweigh the advantages.  "Rename file from properties" (or whatever the new name is) works fine for me.
Logged
John

hit_ny

  • Citizen of the Universe
  • *****
  • Posts: 3310
  • nothing more to say...
Re: Database Optimizations: 12.0.492
« Reply #14 on: May 05, 2008, 10:48:41 pm »

It's just that we're reticent to add anything that would automatically change filenames.  That's why it's an explicit user action now.  (and only really a few clicks to have it check / cleanup the entire library)

Defnitely!

Changing anything as important must be user-initiated.
Logged

zirum

  • Galactic Citizen
  • ****
  • Posts: 403
  • still learning.
Re: Database Optimizations: 12.0.492
« Reply #15 on: May 05, 2008, 11:43:01 pm »

Hi!

This sounds neat. Haven't had the time to test it yet, but might it also be the first steps for a more "relational" approach for Artists as well?  :)
Logged
Note to myself: Read, think, write - Read, think, write - think, read, write - think, write, read - think, write, read... Aahhw, i always mess that up...

Tolga

  • Regular Member
  • Galactic Citizen
  • ****
  • Posts: 438
Re: Database Optimizations: 12.0.492
« Reply #16 on: May 06, 2008, 04:22:52 pm »

I'd like this feature.

It's just that we're reticent to add anything that would automatically change filenames.  That's why it's an explicit user action now.  (and only really a few clicks to have it check / cleanup the entire library)

Yes, I agree. This really shouldn't be automatic. If it is going to be an option, the default should be off.  this would also mess up with the ability to take synchronized copies of mp3 libraries and Filenames don't support all tag characters.
Logged

Tolga

  • Regular Member
  • Galactic Citizen
  • ****
  • Posts: 438
Re: Database Optimizations: 12.0.492
« Reply #17 on: May 06, 2008, 05:14:59 pm »

So, I haven't tried yet but this optimization sounds like it might improve database load times significantly during remote access, which seems to be the current bottleneck of server access.  8)

Great!
Logged

Qythyx

  • Galactic Citizen
  • ****
  • Posts: 390
Re: Database Optimizations: 12.0.492
« Reply #18 on: May 06, 2008, 07:53:57 pm »

So I won't see much benefit with the following scheme?

M:\Music\[Artist]\[Album]\[track #] [Name].wma
Logged

Lasse_Lus

  • Citizen of the Universe
  • *****
  • Posts: 999
Re: Database Optimizations: 12.0.492
« Reply #19 on: May 08, 2008, 03:14:01 am »

..with this DB Optimization, MC feels almost new...browsing around within MC is a pleasure

big thanks  ;D
Logged
MT5FR
Pages: [1]   Go Up