Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[JustinChase]

This has been going on for awhile.

I use Deluge to download new TV shows, then Sonarr automatically moves them to my TV storage location once finished.

I can take days, if ever, for MC to pick up on the new file and import them into MC.  I usually just force a manual Import to get it to happen, which always works.

I don't know what the 'trigger' is for MC to see/act upon a new file, but it's not getting tripped in my case.  Here are some screenshots showing the latest issue.

The first shows when Deluge finished downloading.

The next shows the date Windows has for the file, which is sitting in the correct folder

The last shows Theater View not reflecting this new file.

I'm happy to provide any further information if necessary.

[blgentry]

Since your files are on a NAS, they probably aren't producing file system "events" for MC to trigger it's auto import.  MC should be running auto import every two hours by itself, which might explain the delay you are seeing.  Some details here:

https://wiki.jriver.com/index.php/Auto-Import#Background_Scanning

Brian.

[JustinChase]

maybe, but it's just a mapped network drive.  I don't know how else one would have more than a terrabyte or 2 of data if not on some sort of NAS.

I have the drive mapped as the V:/ drive in windows on the computer running this copy of MC.  If that's a problem, them perhaps the system needs tweaked.  How else could I get this to just work?  It imports everything just fine when I manually run auto import from settings.

[blgentry]

maybe, but it's just a mapped network drive.  I don't know how else one would have more than a terrabyte or 2 of data if not on some sort of NAS.

I have kind of a silly amount of storage directly connected to my computer via USB.  I think it's around 40TB in total, not including the internal drive.  But I recognize that my way of doing things isn't popular.

I have the drive mapped as the V:/ drive in windows on the computer running this copy of MC.  If that's a problem, them perhaps the system needs tweaked.  How else could I get this to just work?  It imports everything just fine when I manually run auto import from settings.

I don't think mapped drives are necessarily a problem.  It's just that not all NAS manufacturers implement "file system events" which tell MC to run auto import.  If it's running every 2 hours by itself, then it's working normally.  If not, then something else is wrong.  You can always add an auto import button to the top or bottom button bars, which gives you really quick access to that function.

Or you could investigate optional settings on your NAS, or perhaps firmware updates that support file system events.  I know very very little about the details of this.  Just throwing out an idea.

Good luck.

Brian.

[Hendrik]

Network drives rely on the remote system to actually provide file system change notifications. A NAS may fail doing that, but ultimately it would be responsible to emit those events - or just don't advertise that it can do that, then MC would schedule a timed import.

This works perfectly if you have a remote windows server, for example. A NAS with its own implementation of the SMB protocol .. who knows. I would inquire with the NAS makers if there is anything to do about that.

[JustinChase]

it's unRAID, which uses SMB.  As far as I know, it just acts like extra storage, which I've mapped to windows.  Not sure why that would be any different than a windows server.

If I run an import, it finds the files just fine.  It does not find the files every 2 hours however.

Why would a manual import find them without issue, but the automatic every 2 hour import not find them?

Something feels wrong with it, but I'm not a networking engineer.

Can I just schedule a 'manual' import every hour or 2?

[Hendrik]

There is no automatic timed import. It relies on file system events (ie. it gets told when something changes), which unRAID seems to not be sending properly. A "correct" SMB implementation would send this, ie. a Windows server would.
We'll ask the server if it can supply notifications, and if it says it does, then we'll rely on that. If it then doesn't actually send those, well.

[blgentry]

There is no automatic timed import. It relies on file system events (ie. it gets told when something changes), which unRAID seems to not be sending properly. A "correct" SMB implementation would send this, ie. a Windows server would.

Is the wiki wrong then?  It says:

Media Center's Background Auto-Import works by:

* Monitoring File System Events for changes and responding instantly to only those files that changed in a watched directory.
* Running a background full-scan of the defined watched directories, and checking for changes, once every two hours. This is run with a low thread priority to prevent it from impacting foreground tasks.

From:  https://wiki.jriver.com/index.php/Auto-Import#Background_Scanning

Brian.

[Hendrik]

It only does the second if the first was not possible, ie. if the file system or remote server does not support events.

[JustinChase]

Thanks Hendrik, I've posted on their forums also

[Krazykanuck]

I have an unraid server and don't have any issues with auto import.
I do all my maintenance and adds from one computer and my HTPC auto imports.

I don't map the drive I monitor my \\unraid1\  which shouldn't make a difference.

[JustinChase]

I believe if I manually move a file around on unRAID, MC picks that up.  it's when I have Sonarr automatically grab the torrent file, send it to Deluge, then Sonarr moves it to the final location automatically that MC doesn't see this new file.

Do you do this also, or is all your work done by manually moving files around?

[MusicHawk]

Another factor that could be a reason for the behavior, and happens even with a local disc, is how promptly (or not) MC updates a view when the contents have changed, such as a smartview of files selected by tags, or an Explorer-type view of all files in a particular folder. Both types of MC views are slow to be updated when a file is imported, manually dragged in, or has tags updated, changes that should cause the file to appear in the view, but...

In my experience over several versions of MC, updating a view based on such changes can take several minutes, and sometimes doesn't seem to happen at all unless the view is exited, or changed to a different top-level view, then back again. Or wait, and wait, and wait. Often there's a notable time gap.

The issue is proven by viewing the underlying folders outside of MC. The new or changed file is there, but MC is not seeing it and/or not updating the view for quite a while, so long that it is often necessary to go look at the folder to be sure the file is really there. This is a big thing for me because I frequently work at the files level (lots of ripping, editing, etc). To maintain sanity, with two monitors, I have a file explorer open on one monitor, and MC on the other, so I know physical reality vs. what MC shows.

But it is not just physical vs. view, it is a delay updating smartviews when tags change. Typically, looking at new files in an MC Explorer view, I add some tags (Artist, etc), then switch to a smartview that should show the new files (often a sub-level of a top view of all Artists). The new track is NOT there even though the file is there and has the necessary tag. Eventually, it appears, but the delay ranges from confusing (is it really there) to annoying (I need to continue working on it, but I don't see it).

Pressing F5 in MC to refresh the view does NOT help. Why not? I realize it would not be wise to have MC constantly checking for changes. But shouldn't it respond to the user?

The MC view eventually catches up with reality, but a view update doesn't seem to be a particular elapsed time, I've waited 5 seconds, a few minutes, or several minutes before the file appears in an MC smartview. Drives me crazy at times. Perhaps view updating is on a fixed (but invisible) internal schedule?

I've reported this a few times as an anomaly but after a few years it remains. Makes sense that auto-refresh is periodic, but ignoring user F5 presses seems like a bug...

My point is, when trying to sort out Import and other drive/folder/file/tags changes, it's tricky to compare what physically happens to what MC displays.

[mattkhan]

Thanks Hendrik, I've posted on their forums also

You probably want to include details like which version you are running, what sort of share the files are on. Fwiw I also use unraid (6.6.3 atm) and MC sees files pretty much straightaway as far as I can tell. Does this affect particular file types or locations or applies to everything?

[JustinChase]

You probably want to include details like which version you are running, what sort of share the files are on. Fwiw I also use unraid (6.6.3 atm) and MC sees files pretty much straightaway as far as I can tell. Does this affect particular file types or locations or applies to everything?

As far as i can determine, it only happens with files that Sonarr has moved to the final location after downloading has finished.  I have my temporary download folder excluded from the import process (to not have incomplete files imported), but the final location of the finished files is where all my videos are stored, so should be imported automatically.

As I stated, if I manually move a file to this location, MC seems to pick it up right away, it's only (as far as I can determine) an issue with files that Sonarr moves and renames once downloading is finished.

I'm not well versed on what sort of events should happen, or why Sonarr moving a file would prevent this, or even if it's a Sonarr thing.

I'll give the unRAID devs a day or so after the weekend to see if they respond, and if not, I'll try adding specifics, but I'm also running the current version, and don't know what else they might need to know to troubleshoot.

[mattkhan]

As far as i can determine, it only happens with files that Sonarr has moved to the final location after downloading has finished.  I have my temporary download folder excluded from the import process (to not have incomplete files imported), but the final location of the finished files is where all my videos are stored, so should be imported automatically.

As I stated, if I manually move a file to this location, MC seems to pick it up right away, it's only (as far as I can determine) an issue with files that Sonarr moves and renames once downloading is finished.

Where is it moving from/to? one user share to another? are they on the same underlying physical disk or different? is it a move or a copy? I would think these are the key points as it would then be a question of determining whether unraid emits the relevant file events in that situation. The Q for the jriver devs are which events are they interested in exactly? I didn't think that inotify was supported by smb/cifs but memory v hazy so could be mixing up a few things there.

I suspect it will be quite hard to get info without that level of detail but you never know.

[Krazykanuck]

I am using RM&C from my maintenance PC to my unraid server.
It looks like you are downloading to your unraid server then moving to your monitored folder.

Why that would be different as far as windows is concerned I don't know.

[JustinChase]

I use Sonarr to send the torrent or nzb to Deluge or SABnzbd, which then download the file to my L:/Downloads/Incomplete folder, which is not monitored.  Once finished, they notify Sonarr, which then renames the files and moves them to my V:/TV/Series/Season xx/ folder, which is monitored.  If it's a Deluge (torrent) file, it leaves a copy in the incomplete folder to continue seeding.  SABnzbd doesn't offer this option, so it's just moving the file.

I only use user shares with unRAID, which can use any of my actual drives.

[mattkhan]

One plausible explanation is that moving a file from one user share to another, when both shares can use that physical disk does not emit any sort of inotify event because the underlying physical file has not been touched. You can see this by logging into the unraid server and doing something like

Code: [Select]

cd /mnt/user/
touch shareA/test
stat shareA/test
mv shareA/test shareB/test
stat shareB/test

you'll see the file has the same inode

I suppose you could test this yourself by adding files and moving them around yourself between user shares on the unraid server as per the above example. You have to do this on the server btw not by copying around from a client machine.

[JustinChase]

I just realized that I do use a cache disc, so it will be on that physical disc until 2am after it finishes, then it will be moved to an array disc. This still does not result in MC picking.up the new file

[Hendrik]

One plausible explanation is that moving a file from one user share to another, when both shares can use that physical disk does not emit any sort of inotify event because the underlying physical file has not been touched.

Moving or renaming a file should still emit a change event, its irrelevant if the file itself changes. We register both for directory changes (to catch new/renamed/deleted files), as well as file changes themselves.
And yes, SMB/CIFS certainly supports that.

There is of course a variety of factors that could break it on unRAIDs side, and its impossible to really know what does. But I imagine they would rely on standard tech like Samba to provide SMB shares, and linux kernel inotify support.

[JustinChase]

Hendrik, is there anything I can do to test this on my server, to help find the underlying issue?  Any special commands I can run or anything?

It surely seems like an unRAID issue, so I don't expect you to spend time on it, but I'm willing to do whatever to help diagnose.

[mattkhan]

There is of course a variety of factors that could break it on unRAIDs side, and its impossible to really know what does. But I imagine they would rely on standard tech like Samba to provide SMB shares, and linux kernel inotify support.

IIRC user shares are implemented using their own custom FUSE based filesystem (as a union of the underlying disks) so it might be a bit muddier than that. A question for the unraid people I suppose

[RoderickGI]

files that Sonarr moves and renames once downloading is finished.

Once finished, they notify Sonarr, which then renames the files and moves them to my V:/TV/Series/Season xx/ folder, which is monitored.

I'm not sure if this has any relevance, but does Sonarr Rename the file first, and then Move it, or Move it and then Rename it?

As Hendrik says, on a normal drive the events would be captured and used to trigger Auto Import, but I could see a scenario where a file is moved to the monitored location, MC detects it, but the file is immediately renamed, and so is either in use or no longer exists, so it isn't imported. Of course, the rename process should also trigger and update by MC, depending on settings, but if it hasn't been imported yet, it can't be updated.

I would suggest capturing the issue in a log and searching for the original file name and the new file name, and see if MC reports anything like file in use, missing, or similar.

[JustinChase]

I'm not sure if this has any relevance, but does Sonarr Rename the file first, and then Move it, or Move it and then Rename it?

As Hendrik says, on a normal drive the events would be captured and used to trigger Auto Import, but I could see a scenario where a file is moved to the monitored location, MC detects it, but the file is immediately renamed, and so is either in use or no longer exists, so it isn't imported. Of course, the rename process should also trigger and update by MC, depending on settings, but if it hasn't been imported yet, it can't be updated.

I would suggest capturing the issue in a log and searching for the original file name and the new file name, and see if MC reports anything like file in use, missing, or similar.

I don't know, but I suspect it simply renames the file from cache/downloaded to v:/TV/Series/Season/newname as one process, so it is probably one step.  but again, I'm no expert.

I'll follow up with the unRAID guys tomorrow evening if I've not heard back by then.

[mattkhan]

I ran that test I mentioned above and it works fine here, MC put the file into the library in under a minute.

[tzr916]

I've had very similar issues on my setup for as long as I've been using MC (years), across all versions, etc... Download videos on my laptop, do a cut/paste to a local sata drive on the Server PC that MC is watching. MC will refuse to see any of the new files, or it will see a few of the new files but 2 or 3 won't be seen. Sometimes I find out more than 24 hours later. So I end up doing a manual Auto Import on the Server.

I've just learned to live with it, created Auto Import button in theater view, and most recently using a shortcut on my laptop for Auto Import via mcws. Still debating if I want to create a Windows Task on my Server to perform MC Auto Import every hour, on the hour, or at least at Midnight (the later being something that MC should have built in and configurable IMO)... The current "silent" method just doesn't work.

[JustinChase]

This is the only response I've gotten so far...

"Unraid is just using a standard Samba implementation for SMB support.    If the events you want are not getting generated then this sounds like a generic Samba bug.   Have you tried perusing the Samba mailing list to see if there is a known problem in this area."

I'm not sure where to even look, nor what to look for, so it's not helpful to me, but perhaps this is just a Samba bug.

I finally had to manually import the most recent file, and saw something interesting when I did so (screenshot).  It seems MC recognized it as a new file AND as an update file in one fell swoop.  Not sure how that could be, but perhaps is a clue to what's happening.

[mwillems]

This is the only response I've gotten so far...

"Unraid is just using a standard Samba implementation for SMB support.    If the events you want are not getting generated then this sounds like a generic Samba bug.   Have you tried perusing the Samba mailing list to see if there is a known problem in this area."

I'm not sure where to even look, nor what to look for, so it's not helpful to me, but perhaps this is just a Samba bug.

I finally had to manually import the most recent file, and saw something interesting when I did so (screenshot).  It seems MC recognized it as a new file AND as an update file in one fell swoop.  Not sure how that could be, but perhaps is a clue to what's happening.

I just thought I'd chime in with one "gotcha" I encountered with Samba and inotify events that caused a similar issue for me.  By default the Linux kernel sets a fairly low maximum for inotify watches.  If you have a lot of directories, a lot of files, and/or a lot of software watching for file changes you may have more things trying to watch than you have allowable inotify watches.  When you exceed available inotify watches, this can lead to unpredictable functioning of software that relies on inotify to identify changes in files or directories, and should show up as error messages in the system journal.  In my case I had non-deterministic failures in auto-import, and also in other software I was using that detected file changes in real-time (file synchronization software) until I fixed the problem, and now auto-import is 100% deterministic (and has been for the intervening years since I figured out this issue).

The fix is trivial on a normal linux system, you just change the kernel parameter for maximum inotify watches to a much higher number (typically in sysctl.conf or similar).  Check out the selected answer here for details on how to fix it and potential consequences: https://unix.stackexchange.com/questions/13751/kernel-inotify-watch-limit-reached

I don't have any experience with UNRAID, but it appears to be Linux based. This may or may not be an issue or easily fixed in that environment.  Just thought I'd throw it out as it took me a good long while to arrive at the issue in my case and it's easy to test and rule it out 

EDIT:
To add a little more info, the default kernel watch limit is 8192.  I have about 150,000 items (files and directories) that MC is "watching", and I set my inotify watches on my NAS to a little over a million, which has worked fine with other software taking out watches too.

[Hendrik]

To add a little more info, the default kernel watch limit is 8192.  I have about 150,000 items (files and directories) that MC is "watching", and I set my inotify watches on my NAS to a little over a million, which has worked fine with other software taking out watches too.

For the record, MC requests a "watch" only on the main folder you have setup for auto-import. At least whats how the Windows API represents that. If some remote system needs to watch every single sub-element below that to be notified about changes, that would be really inefficient, and may result in issues, I suppose.

[mattkhan]

fwiw my unraid system shows

Code: [Select]

$ cat /proc/sys/fs/inotify/max_user_watches
524288


[mwillems]

fwiw my unraid system shows

Code: [Select]

$ cat /proc/sys/fs/inotify/max_user_watches
524288


Ok, if you haven't edited it, then it looks like they changed it to a saner default rather than the kernel default.

For the record, MC requests a "watch" only on the main folder you have setup for auto-import. At least whats how the Windows API represents that. If some remote system needs to watch every single sub-element below that to be notified about changes, that would be really inefficient, and may result in issues, I suppose.

In my case MC was not the only software taking out watches, Unison, and my DE's file indexer were also taking out watches so in my case it was a combination of things.  Many people also report Dropbox on Linux completely exhausting their inotify watches. 

In my case I had the tell-tale error messages about running out inotify watches (once I knew where to look for them!), and changing the setting fixed it for me.  Looks like UNRAID already thought of the issue though and set a saner default.

[MusicBringer]

Another factor that could be a reason for the behavior, and happens even with a local disc, is how promptly (r not) MC updates a view when the contents have changed, such as a smartview of files selected by tags, or an Explorer-type view of all files in a particular folder. Both types of MC views are slow to be updated when a file is imported, manually dragged in, or has tags updated, changes that should cause the file to appear in the view, but...

In my experience over several versions of MC, updating a view based on such changes can take several minutes, and sometimes doesn't seem to happen at all unless the view is exited, or changed to a different top-level view, then back again. Or wait, and wait, and wait. Often there's a notable time gap.

The issue is proven by viewing the underlying folders outside of MC. The new or changed file is there, but MC is not seeing it and/or not updating the view for quite a while, so long that it is often necessary to go look at the folder to be sure the file is really there. This is a big thing for me because I frequently work at the files level (lots of ripping, editing, etc). To maintain sanity, with two monitors) I have a file explorer open on one monitor, and MC on the other, so I know physical reality vs. what MC shows.

But it is not just physical vs. view, it is a delay updating smartviews when tags change. Typically, see new files in an MC Explorer view, add some tags (Artist, etc), then switch to a smartview that should show the new files (sub-level of view showing all tracks by that Artist). The new track is NOT there even though the file is there and has the necessary tag. Eventually, it appears, but the delay ranges from confusing (is it really there) to annoying (I need to continue working on it, but I don't see it).

Pressing F5 in MC to refresh the view does NOT help. I realize it would not be wise to have MC constantly checking for changes. But shouldn't it respond to the user?

The MC view eventually catches up with reality, but a view update doesn't seem to be a particular elapsed time, I've waited 5 seconds, a few minutes, or several minutes before the file appears in an MC smartview. Perhaps view updating is on a fixed (but invisible) internal schedule?

I've reported this a few times as an anomaly but after a few years it remains. Ignoring user F5 presses seems like a bug...

My point is, when trying to sort out Import and other drive/folder/file/tags changes, it's tricky to compare what physically happens to what MC displays.

I agree with you MusicHawk. I find it sooo annoying when the new stuff fails to display itself, until it feels like it. Grrhh.

[MusicHawk]

Reminding that this problem continues, for years, and in 24.0.78. Views do not refresh quickly or reliably.

Further (and really annoying): There's a new file right where MC should see and Import it. It doesn't, so I force Import. Still no file. I check it in  Windows File Explorer, there it it. But THEN, if I open a tab in MC and use its Explorer, which is actually Windows Explorer embedded, the FILE IS NOT VISIBLE. Then when I switch back to my Windows File Explorer window, it takes me back into MC where still, THE FILE IS NOT VISIBLE. Therefore, Windows File Explorer shows all files, UNTIL MC used Explorer, which causes the files to disappear the same as they do in MC views.

Eventually (sometimes an hour later) the files appear.

Why does MC fail to see files? Especially why does it ignore pressing F5? There's no explicit reason, it hides them on its own, then eventually shows them on its own. F5/Refresh does nothing. Breaking Windows Explorer really ruins the day because there's no simple way to decide "Am I crazy, or what"? That can't be "as designed". What is going on???

PS: I installed a different file explorer app which is not affected by MC, so I can reliably see all files. That just confirms MC's annoying behavior, since I can't continue working in MC until it opens the kimono.

This is NOT a NEW problem.