Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[jmone]

I've had issues on and off over the years with something eating memory till the "In Use" memory approaches Total Installed memory.... and I've noticed it starting again.  Tracking these down can be a PITA and while a reboot fixes it (temporarily) it can be inconvenient if you are doing something. 

I really should rebuild this PC and get off the Insider Previews but for now....Here is a very simple AutoHotKey script that I wrote awhile ago to test the memory when trying to track the leak.  One great side affect is that (for me) it actually also releases this lost memory.  All it does is increase a variables memory allocation and when the app closes the "In Use" memory is now greatly reduced.

Attached is the AHK Script and a compiled 64-Bit Version.

[JonnyRedHed]

Wow, thanks. Works really well on my win7 x64 when I've had it on for a few days without a reboot, reclaims tons of memory. 

[JonnyRedHed]

There is also this one.

https://www.ghacks.net/2018/02/27/memplus-memory-management-software-for-windows-review/

[millst]

Free memory is wasted memory. I don't know why so many people think they know memory management better than the developers of OS kernels. Unless you have some specific issue, these things are more likely to slow down your computer e.g. by flushing cached data that has to be reloaded from disk the next time its needed. Even if you have some issue, then you're better off solving the root cause than running some band-aid.

-tm

[RD James]

I see this is an older post now, but I know that you use ReFS drives jmone.
Windows 10 has a habit of eating up all of your memory with ReFS drives - particularly if integrity streams are enabled.

If you use the Sysinternals RAMMap tool, and all of your active memory is being used by "Metafile" then you are affected by this.
The following registry tweaks fix that issue for me:
 

Code: [Select]

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem]
"RefsEnableLargeWorkingSetTrim"=dword:00000001
"RefsNumberOfChunksToTrim"=dword:00000040
"RefsEnableInlineTrim"=dword:00000001

(save as .reg and run)

[jmone]

Yeah - it is an old little app I wrote ages ago when having a mem leak issue with a raid controller.  I dusted it off again when looking at the Mem issues with ReFS and played with those ReFS reg settings last year.  It's now all good (issues started by Windows update and fixed later by another one).  FYI - I'm now also running Windows 10 for Workstations Pro (it includes official support for ReFS). 

[Jonny RedHed 2]

Yeah - it is an old little app I wrote ages ago when having a mem leak issue with a raid controller.  I dusted it off again when looking at the Mem issues with ReFS and played with those ReFS reg settings last year.  It's now all good (issues started by Windows update and fixed later by another one).  FYI - I'm now also running Windows 10 for Workstations Pro (it includes official support for ReFS).

Any 2019 update to your little Memtest. Win10 improvements etc

[jmone]

Nope   .... and thankfully I've not had to use it for ages (no more ReFS Memory Leaks).  That said, I don't see why it would not still work.

[Jonny RedHed 2]

Ok, ta for the reply. Just curious. Works perfect for me using an old low end laptop for a while.

[jmone]

So I started getting the same old issue scanning over millions of items on my Win2016 Server.  RD James's suggestion worked like a charm.

I see this is an older post now, but I know that you use ReFS drives jmone.
Windows 10 has a habit of eating up all of your memory with ReFS drives - particularly if integrity streams are enabled.

If you use the Sysinternals RAMMap tool, and all of your active memory is being used by "Metafile" then you are affected by this.
The following registry tweaks fix that issue for me:
 

Code: [Select]

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem]
"RefsEnableLargeWorkingSetTrim"=dword:00000001
"RefsNumberOfChunksToTrim"=dword:00000040
"RefsEnableInlineTrim"=dword:00000001

(save as .reg and run)

[eve]

So I started getting the same old issue scanning over millions of items on my Win2016 Server.  RD James's suggestion worked like a charm.

No b/s review of ReFS? Is it anything special or just more of a hassle?

Very curious about people still using Windows for their storage needs in 2022. It's not a dig or anything, I'm just interested in what people prefer and how they interact with their systems.

[jmone]

I actually really like ReFS if you:
- are Windows based and have an edition that supports it (minimum is Windows for Workstations, eg not home or pro)
- for storage drives (not for the OS)
- remember to format the drives with integrity streams enable " /i:enable"
- hate fdisk (as you will never need to run it again)
- works great without storage spaces (eg I use Drive Pool over the top to JBOD them into one volume)

I guess I've been running ReFS on my pools for coming up to 10? years.  Apart from the memory mgt weirdness that has come and gone with various Windows Patches/Updates, I've never had an issue with performance, reliability, or resilience.  I ended up on ReFS after my RAID controller and FDISK had a fight completely scrambling my FAT.   Prior to that I was using MS own "Windows Home Server" that did disk pooling but then SKU went away.

These days I've two 100TB ish pools, one on my MAIN-PC running WindowsForWorkstations 11H2, that is also my MC server (as well as everything else I do day to day).  2nd setup is a WinServer2016 that does the backups of all the PC's C: Drives and has a copy of all the media on the MAIN-PC's (I use FreeFileSync to push the media selectively after first previewing the changes) over a 10Gbe fibre link (with SSD caching writes on each pool before they are rebalanced out to spinning rust). 

I've pretty satisfied with this setup as I've now got running on Windows:
- ReFS formatted discs
- Pooled together as one big vol with easy add/remove of individual HDD
- VFast copying between the pools
- Real backup and preview of changes before a commit

[eve]

I actually really like ReFS if you:
- are Windows based and have an edition that supports it (minimum is Windows for Workstations, eg not home or pro)
- for storage drives (not for the OS)
- remember to format the drives with integrity streams enable " /i:enable"
- hate fdisk (as you will never need to run it again)
- works great without storage spaces (eg I use Drive Pool over the top to JBOD them into one volume)

I guess I've been running ReFS on my pools for coming up to 10? years.  Apart from the memory mgt weirdness that has come and gone with various Windows Patches/Updates, I've never had an issue with performance, reliability, or resilience.  I ended up on ReFS after my RAID controller and FDISK had a fight completely scrambling my FAT.   Prior to that I was using MS own "Windows Home Server" that did disk pooling but then SKU went away.

These days I've two 100TB ish pools, one on my MAIN-PC running WindowsForWorkstations 11H2, that is also my MC server (as well as everything else I do day to day).  2nd setup is a WinServer2016 that does the backups of all the PC's C: Drives and has a copy of all the media on the MAIN-PC's (I use FreeFileSync to push the media selectively after first previewing the changes) over a 10Gbe fibre link (with SSD caching writes on each pool before they are rebalanced out to spinning rust). 

I've pretty satisfied with this setup as I've now got running on Windows:
- ReFS formatted discs
- Pooled together as one big vol with easy add/remove of individual HDD
- VFast copying between the pools
- Real backup and preview of changes before a commit

Okay, you've got this figured out. This is cool af and really pushing Windows to it's max. Look man, I'd be tearing my hair out here, the death of small companies running microsoft exchange / domain servers internally was a massive sigh of relief on my part. I *hated it*. So it's really sweet to see someone rocking Windows servers, in their home, in a comfortable way that works for them! That's what I *wanted* Windows server to be.

I'm Windows based on the machines I actually sit at, but the majority of my infrastructure / backend including serious storage is linux.

So what you've done is gone with disks formatted as ReFS but then pooled them on your own with DrivePool? Very, very smart. I'm a big advocate for JBOD pooling in non critical situations. RAID is... not a backup, and it's less useful than it used to be beyond extremely high availability situations where downtime is unacceptable.

I'm all about JBOD and intelligent pooling here. I use MergerFS with spinning disks, very fast cache SSDs and enterprise cloud storage. It's stunningly useful.

Can you explain this "Preview Of Changes Before A Commit" ? Are we talking about when backing up between your 2 arrays?

[jmone]

So "FreeFileSync" offer a "compare" feature that shows you want will be copied, deleted, or updated from one pool to another.  This lets you visually check that the changes that you are about to make are what you want and helps prevent unended modification on you back up pool.  This has saved me a few times where the compare has shown I'd accidently deleted some content and I could then just copy it back.  It also shows when I have have stuff I want to delete but have not gotten around to eg (eg: TV Records I'd made for testing etc).

I'm only running "Windows Server Essentials 2016" as it is the last of the "Server" editions that still offers non-domain, cluster based, incremental, bare metal & file based backups at a reasonable price for home use.  Otherwise I'd just spin up a Windows for Workstation box with ReFS / DrivePool / Free File Sync.  No 2nd OS to learn the foibles of.  I should say this backup PC is all made with hand me down parts from my MAIN-PC when it gets upgrades (Mobo, RAM, HDD etc) all mounted in a cheap 20 bay case and e-bay sourced "server" bits for the SAS Controller / 10Gb SFP+ cards and adapters.  Hardest bit was flashing the IT firmware on IBM M1015 SAS cards to which I then added some random SAS Expander to run the (up to) 16 spinning HDD.  One rant is that the consumer Gbe interface has no driver support if it is intel based, so I had to also add a USB Ethernet adapter to get WOL (no WOL on fibre) as I also run "Light Out" that keeps the server in standby unless it is needed for backup or if I wake it up to push a media sync.

I do miss the plug and play simplicity of my first "Windows Home Server Box" that could take 10 drives and was all prebuilt (I had the add on that gave a 2nd 5 bay expansion option).