Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Captain_Bitter]

First off, I'll admit I have a massive music library. My issue is I can't understand the disk space that thumbnails take up.

This week, I am rebuilding my entire library from scratch (don't ask) on a Mac Mini with a small hard drive (250GB). The actual library itself seems a reasonably small size, but the thumbnails.....

When I perform a ^i on the thumbnails, I get the following information:

Size: 1,335,560,747 bytes (62.95GB on disk) for 10 items. And this is still synching so is climbing gradually.

This is 1.3GB worth of thumbnails, but it takes up nearly 63GB in space! And the available disk space is dropping so it appears that the disk space used is "real".

FYI: I have between 1.2-1.3M tracks in my collection, so I expected the thumbnail creation would take 1-2GB. What I can't explain/understand is why the disk space is 70GB+ when the files themselves only seem to be taking up 1.3GB.

Can anyone explain how this works and/or why the space demands are so high? Or - as I suspect - is this some sort of glitch where the disk size used is climbing unnecessarily? If the latter, what can I do to fix this? 70GB or more will eat up most of my available disk space and I won't be able to load any more apps.

Thanks in advance, Dave

[zybex]

That's a huge difference. The real space is the 1.3GB, the rest is perhaps temporary/cache/snapshots stored by the OS itself. While rebuilding thumbnails, MC is constantly rewriting the Thumbnails DB files which perhaps causes the OS to automatically generate Snapshots (backups) of the previous version of the file. You can clean these snapshots manually, see links below.

If it's an SSD, it's possible that the OS will eventually do a TRIM to recover the space once MC stops writing to the file. This "delayed cleanup" is a performance optimization.

Or Purgeable Storage?
https://osxdaily.com/2021/04/22/what-clear-purgeable-storage-mac/

APFS snapshots?
https://www.reddit.com/r/MacOS/comments/11st1aq/just_recovered_60_gb_of_missing_capacity_by/

[Captain_Bitter]

Thanks for the response, zybex. Yes, something weird is going on here, and it may indeed by MacOS-related. I decided to go the reboot route, so in doing so thought I would install a pending Sonoma update (14.3). Lo and behold - and before the machine restarted -- now my free disk space is showing 101GB free. However, when I go over to the thumbnails, I still see: 1.47GB used, 76.75GB on drive.

Soooooo:
1. it looks like the space was freed appropriately, but....
2. The space still shows incorrectly via "Get Info" (^i).

Really, really weird. I am rebooting now and will see what happens when the system starts back up.

Dave

[Captain_Bitter]

That was a weird MacOS bug. I'm assuming it was MacOS, anyway.

Upgraded to Sonoma 14.3 and rebooted. After having some initial problem with Finder restarting, now the # of bytes and the storage space are in alignment. Whew -- at least I have a bunch of usable space back.

Thank you for your help. Zybex, the article you pointed me to was not the issue, but in that article they suggested rebooting (and updating if you had an update pending....which I did), and that seems to have solved the problem. At the same time, I also had a playback issue where the sound was muffled and distorted on all tracks, and that seems to have vanished, too. Odd.

And now, while I am updating more thumbnails, the problem is back in "Get Info".....but it doesn't seem to be impacting the actual disk space in Finder. Curiouser and curiouser.


Life is a mystery, Dave

[Captain_Bitter]

Nope -- spoke too soon. The issue persists with the disk space getting eaten up. At this point, I think i'll just repair the thumbnails a little at a time, figure out how to recover the disk space and start back on the thumbnails. Just do this piecemeal.

[MrBiff]

These may start to help explain disk usage wrt. Snapshots:

https://eclecticlight.co/2022/02/10/managing-snapshots-how-to-stop-them-eating-free-space/
https://eclecticlight.co/2022/02/13/last-week-on-my-mac-still-struggling-with-snapshots/
https://eclecticlight.co/2022/07/23/explainer-the-arithmetic-of-snapshot-size/

and macOS + SSD TRIM:

https://eclecticlight.co/2023/04/14/will-your-new-ssd-support-trim-and-smart/

[JimH]

These may start to help explain disk usage wrt. Snapshots:

https://eclecticlight.co/2022/02/10/managing-snapshots-how-to-stop-them-eating-free-space/
https://eclecticlight.co/2022/02/13/last-week-on-my-mac-still-struggling-with-snapshots/
https://eclecticlight.co/2022/07/23/explainer-the-arithmetic-of-snapshot-size/

and macOS + SSD TRIM:

https://eclecticlight.co/2023/04/14/will-your-new-ssd-support-trim-and-smart/

Wow!  Thanks!  Explains a lot and may account for many of the very strange Mac problems we've seen for a while.

[Captain_Bitter]

The file that "bloats" is: Thumbnails (large).jmd. All the other files in the Thumbnails folder take up the same size as their bytes count.

Currently, my Thumbnails (large).jmd file shows as:  1,619,001,334 bytes (23.4GB on disk). The (medium) and (small) .jmd files are size-correct.

MacOS has corrected this on occasion, reducing the disk space to conform to the byte count. However, I have not been able to do anything on my part to initiate this. It just happens when it happens.

If anyone has any ideas on how I can initiate this 'size-correcting' process and recover the space myself -- instead of waiting for MacOS to do this -- I'd be most appreciative for suggestions.

Thanks, David 

[MrBiff]

I'm afraid you're not going to like reading more about this...

https://eclecticlight.co/2023/04/17/the-finder-confuses-with-wildly-inaccurate-figures-for-available-space/

[Captain_Bitter]

Happy Happy, Joy Joy. Thank you for the article. You are correct, not what I wanted to "hear," but I appreciate the knowledge transfer.

Further making this behavior strange, I went in to Systems Settings > General > Storage and, when I did a mouse-over on the System Storage, the extra storage taken up disappeared. You'd think this would be easy for Apple to implement, but apparently not. 

Well, at least I got my space back.......for now.

[MrBiff]

Providing Free space information for a copy on write file system is very challenging.

There are more relevant articles at the site.  Search Finder.

[Captain_Bitter]

OK, so a couple updates:

1. This is definitely a bug. Not necessarily a JMRC-initiated bug, but a bug with how JMRC interacts with MacOS.
2. This bug impacts both 'Thumbnails (large).jmd' and 'Thumbnails (medium).jmd'. At first, I thought it only impacted the 'large' file, but that's only because the 'medium' file is smaller to begin with....so the file bloat doesn't impact the totals as much. But it definitely is both files impacted.
3. The bug causes the (in my case) 2GB thumbnails (large) file to take up an increasingly big amount of disk space. Creating 100k thumbnails drove this file to 75GB, and the applications storage (seen in System Settings > General > Storage) to 125GB!!
4. The thumbnails files seem to be the only files impacted.
5. Once the storage problem is created, all I can do is wait until the OS retrieves the errant disk space. I thought I could do something by going to Storage, but I was incorrect on that.
6. Although I have an older version (JMRC27), unless thumbnails are created differently in subsequent versions (doubtful), this problem is likely to persist up until the latest version.
7. This may be a MacOS issue, but it does not impact other apps that I have seen. I am demoing another music playing app right now, and this effect is not seen in that app -- file sizes in bytes and the disk space requirements are identical.

For me, it means I need to create as many thumbnails at a time as my disk space can handle, stop, quit JMRC and wait for a "cooling off period" for MacOS to recover the disk space.....then restart thumbnail creation. I estimate I will have to repeat this process at least 20x to get all the thumbnails created. And, once I do, I better not have cause to recreate the database or else this problem repeats itself. Once I let the disk space get down to about 20GB free (too low), everything slows down on this system. I have yet to determine how was down I can drive the disk space before performance is impacted.

I'd be interested if anyone else has seen this issue with JMRC. Additionally, if anyone on the development side is planning to look into this, I would be happy to provide database files, etc. for testing/debug purposes. At present, I have been waiting about an hour for the "incorrectly gobbled up disk space" to free up, with no end in sight. So creating all the thumbnails may take me well past this weekend, assuming I am diligent about checking the disk space regularly.

Regards, Dave

P.S. Looking into the APFS Snapshot issue, when I go into the Disk utility, the option to "View Snapshots" is greyed out. I believe that means I only have one snapshot, so it is unclear if this is part of (or related to) the issue/bug I have outlined above. If it is related, having one Snapshot means I can't do anything to correct this issue by deleting previous Snapshots, so this isn't a solution for me.

[Captain_Bitter]

Well, this frees up the "frozen" space.

Reindex Apple Spotlight

When the Apple Spotlight index is corrupted, macOS may display the free space improperly in Finder. To get rid of this issue, you can reindex Spotlight with The Terminal command. This solution also works when you can't search an external hard drive on a Mac.

Choose Apple menu > System Settings.
Click Siri & Spotlight in the sidebar, then scroll down and click Spotlight Privacy on the right.
Click the add button (+) and select the disk to add.
Select the disk that you just added. Then click the remove button (–) to remove it from the list.
Click Done, then quit System Settings.
The drive's contents will be reindexed by Spotlight. Depending on the volume of information being indexed, this can take some time.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------

I can attest that this works. Once the drive was reindexed, I got the space back right away.

[JimH]

This is probably an Apple bug.

Your steps above probably cause the OS to do "garbage collection".

[Captain_Bitter]

This is probably an Apple bug.

Your steps above probably cause the OS to do "garbage collection".

No doubt. But the question wrt JMRC is: what is it about Thumbnail creation that causes the MacOS "garbage collection" (or "garbage creation") to go off-the-rails? It doesn't happen when loading files. It doesn't happen when playing. It doesn't happen with any other app I use on this machine.

So there *IS* something about Thumbnail creation that causes this issue....in conjunction with a likely MacOS issue. What I am wondering is if there something about Thumbnail creation that could be modified so the problem doesn't occur. For example, is Thumbnail creation designated as purgeable/non-purgeable space within the app? I don't know what the answer is, merely asking if this could be done in a different way.

Note: even after the previous referenced action to recover space, when I look at the Thumbnails folder, it still shows 2.95GB....and nearly 74GB "used".

[zybex]

This is a feature of the OS/Filesystem and there's nothing the app can do to disable it - at least I found no flag in the documentation that an app could use to opt-out of it (there's INODE_SNAPSHOT_COW_EXEMPTION, but I found no way for an app to set it)

The APFS snapshots are like automatic backups; each time an app modifies an existing file, the current version of the file is preserved (the "copy-on-write") in case the user wants to revert to it. Time Machine uses these local-backups to generate a full snapshot for external backup.

When MC is creating the thumbnails file it necessarily does so in batches - open file, add some thumbnails, close. Each time you get an additional copy of the previous file stored by the filesystem. Any app constantly rewriting its own files in this matter will trigger the issue, but it's hardly the app's fault. The OS should provide a flag to opt-out of the behavior, for instance by allowing to tag the file as a Cache/Temp file (maybe there's a way, I just couldn't find it on a quick search).

The good news is that you really should not care about this - the OS should automatically recover this purgeable space when you actually need it (at least in theory).
This thread adds some details and lists a couple of commands you can use to list and cleanup these backups:
https://news.ycombinator.com/item?id=38874150

Another option - why not replace your 256GB SSD/M2 for a 1TB one? They're cheap now (don't buy from Apple!), but will likely increase in price this year.

[EnglishTiger]

Another option - why not replace your 256GB SSD/M2 for a 1TB one? They're cheap now (don't buy from Apple!), but will likely increase in price this year.

Unfortunately this may not be possible on a Mac with a M1 or M2 chip.
Although Apple are very good at letting prospective purchasers know the M1, M2 & M3 can handle SSD's up to 8TB, its only when they go to compare them or purchase one that they discover that Apple only build and sell M1 and M2 equipped devices with either a 256 or 512 mb system disc; and there is only 1 high-end model from the M3 range that is available with a 1TB system disc.

[zybex]

"Apple has soldered the SSD (solid state drive) storage onto the Mac mini's logic board, meaning that you can't remove it and plug in a new one."

1TB SSD storage   + $400.00

... of course they did

Meanwhile in the PC world:
 Crucial P3 1TB Internal SSD PCIe Gen 3 x4 NVMe  --  $63.99

[Captain_Bitter]

I am still plagued by this thumbnails size issue. I upgraded to MC32 for Mac, but the problem persists. I understand that this is likely to be due to an Apple/MacOS bug. But here's where I find myself now:

Thumbnails (medium).jmd shows in Finder as 1.17GB.....but also shows it takes 59.02GB on disk!! I can see the file bloat here, but even if this excess 58GB is purgeable, MC32 doesn't act like it is. I get all sorts of issues, when I try to reopen MC32, it re-opens only in read-only mode....reflective of the fact that MC32 *THINKS* there is no hard drive space left. It's one thing if finder is incorrect, but it does impact how the app works. Or, in my case, doesn't work.

So I've found where the problem is manifesting, it is either 'Thumbnails (medium).jmd' or (sometimes) 'Thumbnails (large).jmd'. The only way I have found to recover the space is to reboot, which means every time I want to edit some file metadata, I have to do a little and then reboot. This is not viable long-term solution.

Question: is there any way to "right-size" this thumbnails file without rebooting? Is that my only option? I had been using Onyx to free up space, but I have yet to find a maintenance setting that frees up this space without requiring a reboot. So not much help from Onyx, either. Looking for a solution that doesn't require a reboot.

Thanks for your help, -dGB

[JimH]

You can delete thumbnails and MC will rebuild them.