Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[RoderickGI]

That was a much more reasonable and enlightening post Rolf. Thanks. I am surprised that MC reads the whole file, or large parts of it.

However Mattkhan's post does point to a real issue with using FILE_FLAG_RANDOM_ACCESS, and Microsoft warns against using it. https://docs.microsoft.com/en-us/troubleshoot/windows-server/application-management/operating-system-performance-degrades  That article refers to the Windows caching that I was talking about, and that seems to be a problem when using FILE_FLAG_RANDOM_ACCESS. It doesn't refer to memory paging, but I assume that in low memory situations cached data may be paged out to disk, and that might add to the problem, and hard drive load.

There is still a large potential problem with opening and closing a thumbnail database many times to read each thumbnail, even with read-ahead caching of images. Maybe that would be worse than using FILE_FLAG_RANDOM_ACCESS for most users, with smaller libraries.

Maybe something can be done to improve the situation. Maybe this affects some of the other MC processes.

I'll just leave this here.
https://stackoverflow.com/questions/24298878/access-file-using-file-flag-random-access-with-a-limit-on-cache-size

[Hendrik]

I did some additional research on the random access flag and decided to remove it from the thumbnails reader since that file can grow too large for letting Windows cache it.

We don't use the random access flag often, so it was used intentionally where ever it is used, and the only remaining place now is the actual library database files, which are several orders of magnitude smaller then the thumbnails cache, and performance on these is very important, so those are keeping it.

[rolf_eigenheer]

I did some additional research on the random access flag and decided to remove it from the thumbnails reader since that file can grow too large for letting Windows cache it.

We don't use the random access flag often, so it was used intentionally where ever it is used, and the only remaining place now is the actual library database files, which are several orders of magnitude smaller then the thumbnails cache, and performance on these is very important, so those are keeping it.

Hi Hendrik.
I would like to check this with a trial version.
I can imagine that a buffer holding the image data may still be required.


I see relatively simple possibilities for optimization, each of which can bring about an improvement.

a) The reading process, which reads only a few bytes for each thumbnail, is in fact the construction of an index. If you save this index, you don't have to build it from scratch every time.

b) During this reading process, the data are read in ascending order. The JMD file should be opened correctly with FILE_FLAG_SEQUENTIAL_SCAN. Windows can thus increase the throughput through read ahead and, above all, discard the part that has already been read from the buffer.


In a second run, MC then only requests the thumbnails actually required in the view. If we open the file with FILE_FLAG_RANDOM_ACCESS and are lucky, Windows can then keep it in RAM.

If we don't want to rely on luck, we read the required data in ascending order and let Windows optimize the data throughput with FILE_FLAG_SEQUENTIAL_SCAN.
The read data does not remain in the RAM. It is then up to the application to buffer the required thumbnails. But as shown, a few MB are sufficient for this.

[zybex]

Thanks Hendrik!

Kudos to Rolf for keeping up this investigation until a root cause was (apparently) found, leading up to a potential huge performance improvement on large datasets.
If you read the whole thread (like I did now), you'll see that around #20 Rolf was already on to the actual root cause, and persisted on it even after several rebuttals. His understanding of how an OS does file caching under limited memory conditions is mostly correct.

[RoderickGI]

Well, it was a bit of a bumpy road, with a few detours, but persistence does pay off. As above, thanks Rolf.

It will be interesting to see how much this improves (reduces) memory usage. Unfortunately on my systems with my Library and Thumbnail database sizes, it probably won't make a difference.

From what I have read the Windows Cache Manager will automatically turn on the FILE_FLAG_SEQUENTIAL_SCAN if the file is read in a sequential manner, and then turn it off when the read is non-sequential. That seems to be the best of both worlds, as turning it on manually means the Windows Cache Manager will use double the memory for look-ahead (Prefetch) as it would without the flag. But whether it is appropriate or not I can't really say.

[rolf_eigenheer]

It will be interesting to see how much this improves (reduces) memory usage. Unfortunately on my systems with my Library and Thumbnail database sizes, it probably won't make a difference.

This IO overload never occurs where there is sufficient unused RAM. As a result, in these cases simply omitting FILE_FLAG_RANDOM_ACCESS cannot noticeably improve. On the contrary! It must be checked that this does not result in a slowdown in image creation, for example with HDDs.
Maybe you could try this out with a CommandLineSwitch. Because if it turns out that the thumbnail buffer is required when scrolling, then this would first have to be implemented by the application.

But I am glad that the root cause of large memory load is recognized now. This way, you can keep an eye on this in the future.

[RoderickGI]

This IO overload never occurs where there is sufficient unused RAM. As a result, in these cases simply omitting FILE_FLAG_RANDOM_ACCESS cannot noticeably improve. On the contrary! It must be checked that this does not result in a slowdown in image creation, for example with HDDs.

I agree. I shall see if there are any noticeable effects on my 16GB HTPC with the ~38,000 file Library.