Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[JustinChase]

If I select several hundred gigabytes of videos and perform a Rename task, which results in actually moving the files (which I do not infrequently), MC will not respond or be available for anything until it's done moving the files, which can take hours.  It's very frustrating!

PLEASE, make this a true background task, and let me continue to use MC while this is happening.

PLEASE.

[Vocalpoint]

If I select several hundred gigabytes of videos and perform a Rename task, which results in actually moving the files (which I do not infrequently), MC will not respond or be available for anything until it's done moving the files, which can take hours.  It's very frustrating!

Why would you be using MC to move hundreds of gigabytes of files anyway? Move them via Explorer or some other tool and then just run a manual MC import - which should automatically update the the library to reflect any external changes.

VP

[glynor]

Why would you be using MC to move hundreds of gigabytes of files anyway? Move them via Explorer or some other tool and then just run a manual MC import - which should automatically update the the library to reflect any external changes.

This is convenient for a bunch of reasons, but one is that if you are consolidating from multiple "incoming" directories into the "proper" long term storage location.

Plus, reimporting risks losing essential tag data that may include things like [Date Imported].

If I select several hundred gigabytes of videos and perform a Rename task, which results in actually moving the files (which I do not infrequently), MC will not respond or be available for anything until it's done moving the files, which can take hours.

This is NOT my experience at all.  I regularly use Rename, Move, and Copy to move HUGE sets of files, and MC remains usable and responsive while it completes.

There do seem to be some reliability issues right now, but that's just a bug that hasn't been quashed yet.  I've used it this way for years without issue.

Now, I almost always use it to Move files, not Copy. Are you creating copies?

When MC shows the standard Windows file copy dialog, then it does become non-responsive while the copy completes, but moving/renaming does not.  It just queues a tagging change like any other tagging change.

[JustinChase]

Are you creating copies?

When MC shows the standard Windows file copy dialog, then it does become non-responsive while the copy completes, but moving/renaming does not.  It just queues a tagging change like any other tagging change.

No, I'm renaming the files, which moves them into a new location.  If I don't change drive letters (i.e. keep them on the V: drive for example) it's very fast.  However, if I change them from the G: drive to the V: drive it takes a long time.

I wrote that last night after renaming/moving only (4) 3-5 Gb files, and after MC had been unresponsive for about 15 minutes.  It took almost another 45 minutes before I could use MC again.  If I clicked around, it would register the click about every 5 minutes or so, so it's not completely unresponsive, but it's functionally useless during this time.

I'm also connected via wifi, which I know is not ideal, and is likely part of the issue; but still, it seems MC could maybe put this menial task into a true background task, and not hang the whole program until the rename/move is finished.

[JimH]

If you're moving Gb of files and one of the drives isn't local, it should take a long time.

A virus checker looking at every file could also cause trouble.

[glynor]

I wrote that last night after renaming/moving only (4) 3-5 Gb files, and after MC had been unresponsive for about 15 minutes.  It took almost another 45 minutes before I could use MC again.  If I clicked around, it would register the click about every 5 minutes or so, so it's not completely unresponsive, but it's functionally useless during this time.

I'm also connected via wifi, which I know is not ideal, and is likely part of the issue; but still, it seems MC could maybe put this menial task into a true background task, and not hang the whole program until the rename/move is finished.

I'm pretty darn sure that MC is currently doing those operations in a background thread.  Like I said... Mine remains completely responsive when doing renames via the Rename, Move, and Copy tool, including going from physical disk to physical disk.

So... Something else is going on with your system.  A simple test is this:  Does just MC have performance problems while the renames are happening, or does the whole system slow down?  If you start the rename, and then minimize MC (or switch away from it to something else), the rest of Windows should continue to feel snappy.  If not, then the problem isn't process priority in MC, it is that your system is suffering under the load of the rename.

Are you typically, when moving things, moving them from the C drive (or some other partition on the same physical disk) over to a network drive?  I'm thinking the disk thrashing is causing a general system performance slowdown.  In other words, if your system performance over all is being limited by the availability of the C drive, then you could be seeing problems simply because the drive is reading from C a lot and so read performance for other areas on the C drive (maybe your swap file) is falling through the floor.

Two things, if true, would indicate that you are just hitting a disk performance issue:

1. Are you RAM limited?  (2GB total RAM or less, or more with less free due to background processes that launch at boot.)
2. Are you moving from the system drive to somewhere else (or the reverse, but this seems less likely)?

I ask about RAM because if you are RAM limited, then Windows is going to rely on the swap file a lot more.  If you need to use the swap file on the same disk that is being thrashed by a big read operation, then the disk will spend most of its time just seeking back and forth, getting very little "real" work done.  And, since your RAM is effectively an order of magnitude slower, the system itself will grind to a crawl.

These problems would be exacerbated if you are on a laptop with a 2.5" drive.  These are typically very slow drives generally, and worse, they often perform very poorly under a load with high queue depths (when you have lots of separate processes trying to access different parts of the drive simultaneously).

The only other thing I can think of that could be the cause (this would be more likely if you are using a desktop system and NOT renaming from the C drive):  If it is an AMD based system and you have the system vendor's AHCI drivers installed, you should probably uninstall them and just use the Microsoft drivers.  Until the most recent chipsets/CPUs (llano or newer) AMD has serious problems with AHCI performance using their drivers.  And, just like described above, they manifest with terrible performance under loads with high queue depths.

Microsoft's default AHCI drivers don't suffer from this because they emulate the broken parts of AMD's implementation in software.  This trades CPU and RAM utilization for performance, but the tradeoff is worth it.  AMD's AHCI is just plain broken.  Unfortunately, if you're using RAID, you can't switch to Microsoft's RAID drivers.  My only recommendation there would be to buy a real RAID controller, as you'll hit the same problems and you can't work around them by using Microsoft's drivers.

[glynor]

A virus checker looking at every file could also cause trouble.

That's a good point.

Exclude your media storage location in your Virus checker.  At least the network one, if not both "sides".

[JustinChase]

Does just MC have performance problems while the renames are happening, or does the whole system slow down?  If you start the rename, and then minimize MC (or switch away from it to something else), the rest of Windows should continue to feel snappy.  

Only MC.  Everything else on the machine functions just fine.  i can browse, read or create emails, etc with no issues.  MC is the only program that becomes unresponsive.

Nothing is/was on my local C: drive.

I keep everything on an unRAID server.  I have several user shares

J: (Backup)
K: (Documents)
M: (Music)
P: (Photos)
V: (Videos)

When I download a F1 race (for example) it ends up being stored on J:\newsgroups\Canada\Race.mkv

I keep my long-term storage of videos on my V: drive.

So, I "Rename, Move & Copy files..." on the 3.3 Gb file located at J:\newsgroups\Canada\Race.mkv and it gets changed to V:\Racing\Formula 1\Season 2012\Formula 1 - s2012e07 -  Race.mkv

This will take a long time before MC becomes responsive again.

If I then move the same file from V:\Racing\Formula 1\Season 2012\Formula 1 - s2012e07 -  Race.mkv to V:\Racing test\Formula 1\Season 2012\Formula 1 - s2012e07 -  Race.mkv it takes 13 seconds before MC becomes responsive again.  Notice that it's staying on the same V: drive in this case.

Now, if I move it again from V:\Racing test\Formula 1\Season 2012\Formula 1 - s2012e07 -  Race.mkv to J:\Racing test\Formula 1\Season 2012\Formula 1 - s2012e07 -  Race.mkv the windows waiting circle spins for 5 minutes before a click to a new selection in my view is registered/shown by MC, but MC is still unresponsive to any further input, and, in fact, I can't even get MC show in windows.  alt-tab or clicking the icon both result in no action, and MC not being visible.  After another 4 minutes MC finally displayed and let me click another selection in the view, but still has not displayed the results of that selection; just he spinning circle.

Another 5 minutes before I saw the results of the selection, and MC is once again not available to be viewed yet.

Another minute and MC is viewable, but with the spinning circle again, and the view took another 6 minutes for the view to be updated.

It's now been 34 minutes since I moved the file and MC is still not responsive.

I've read and written a few emails, browsed several websites, navigated thru windows explorer, and updated an Excel worksheet all while waiting for MC to become responsive.  Everything else was basically fine, perhaps slightly slower than usual, perhaps not; hard to say.

Avast icon is not spinning, so it's not scanning anything, so I doubt it's any virus scanning causing any issues.  Especially since moving from V: to V: is very fast, but moving from V: to J: isn't.

Any other suggestions what to try to help identify and rectify this very annoying issue?

[Vocalpoint]

I keep everything on an unRAID server.  Any other suggestions what to try to help identify and rectify this very annoying issue?

What's the story with the server? It's drives? If the drives are standard home discs (not enterprise class) or even worse - connected to a standard consumer grade motherboard and all are sharing the same bus on the box - the bottleneck could be  happening right on the machine itself. Especially when moving things between drives.

Some of the slowest file moves I have seen here are "interdrive" moves on the same box.

Let us know what the server has for horsepower...

VP

[JustinChase]

Well, unRAID is different than other servers, in several ways.  However, notice that I said I have several user shares, not drives.  The shares all span all the drives, so it's not necessarily moving from one physical drive to another drive; although it's very likely.

Here are the server specs...

Case: Rosewill - RSV-L4000 -  4U Rackmount Server Chassis 8 Internal Bays, 7 Included Cooling Fans
Power Supply - XFX - Core Edition PRO550W - 80 PLUS BRONZE Certified Active PFC
Motherboard - Foxconn - ELA - LGA 775 Intel G33 ATX
Processor - Intel - Pentium E5200 - Wolfdale 2.5GHz 2MB L2 Cache 65W Dual-Core
Cooler - Rosewill - RCX-Z300 - 92mm CPU Cooler
Memory - Patriot Viper 4GB (2 x 2GB) DDR2 800
NIC - Intel - EXPI9301CTBLK - 10/ 100/ 1000Mbps PCI-E
Hard Drive - (4) Hitachi - Deskstar 0S03230 - 3TB 5400 RPM 32MB Cache SATA 6.0Gb/s
HDD Controller Card - Rosewill RC-218 - 4xSATA II

Also, notice that if I move from one share to the same share it took 13 seconds for MC to become responsive again.  Moving from one share to another share has now taken 1 hour exactly, and MC is still not responsive.

Also notice that everything else on my machine is fine; it's only MC that is exhibiting problems when moving files.

[glynor]

Avast icon is not spinning, so it's not scanning anything, so I doubt it's any virus scanning causing any issues.  Especially since moving from V: to V: is very fast, but moving from V: to J: isn't.

Any other suggestions what to try to help identify and rectify this very annoying issue?

I wouldn't be so quick to count Avast out.  There's a difference between a "scan" (where it would show the spinning activity) and on-demand monitoring.  And, when you move from V to V, it is just renaming the path (the file isn't read), so that wouldn't trigger an on-demand scan.

Disable it completely (or uninstall it just as a test) and try again.  If the problem goes away, or decreases dramatically, then you have your culprit.  I will say, I have absolutely seen similar things with Anti-Virus in the past.  I have my main media folders on my M drive (the big RAID) excluded from MSSE, and I have the JRiver folder in the AppData directory excluded.  Everything is MUCH better behaved that way.  If you uninstalled it completely, and then it acts the same, well then... It is completely ruled out as the culprit.

Worth checking.

Otherwise, I agree with VocalPoint.  We need to know more about the system.  CPU, RAM, OS, and describe the setup on that RAID?  Where is MC's Library stored?  What other background processes might you have running that might be monitoring the filesystem?

unRAID can be VERY slow, and also doesn't handle high queue depths extremely well, so... I agree with VocalPoint.  If you are moving from one volume on the unRAID to another volume on the same unRAID box, you might be hitting problems there (especially if your Library is stored there).

But I can confirm, something is not "normal" there for you.  I can (and do regularly) use Rename, Move, and Copy files to move very large sets of files from my T drive (temporary "TV recording" space) over to my M drive (long term media storage) and it is fine.  I also use it regularly to move from both T and M over to external drives at X, and that also works fine (as far as performance).  Literally, I can't tell on the system that the Move is running in the background except that I see the "Tagging Changes" queue counting down in the lower status bar.  MC otherwise remains "normal" as far as responsiveness.  Maybe Thumbs in views build a little slower and stuff, but nothing where I would call it unpleasant to use, much less unusable.

[JustinChase]

I wouldn't be so quick to count Avast out.  There's a difference between a "scan" (where it would show the spinning activity) and on-demand monitoring.  And, when you move from V to V, it is just renaming the path (the file isn't read), so that wouldn't trigger an on-demand scan.

Disable it completely (or uninstall it just as a test) and try again.  If the problem goes away, or decreases dramatically, then you have your culprit.

I'll do that once MC becomes responsive again.  I was hoping I have logging turned on, and I can provide a log of this 65 minute problem, but I'm afraid killing MC will kill the log also.

I think I'll go make some lunch, clean the pool, then come back and see if MC is willing to work with me again, then will uninstall Avast, and try again.

Any other thoughts/ideas before I try this?

Thanks again for all the help/advice.

[glynor]

Hard Drive - (4) Hitachi - Deskstar 0S03230 - 3TB 5400 RPM 32MB Cache SATA 6.0Gb/s

Ooh... Is one of those the system drive?
On the third-party RAID card?

If so, get the C drive on a 7200rpm drive (worst-case, SSD is much better) and attach it to the Intel controller, not some junky third-party controller.

You do NOT want your C drive on a third-party controller, unless you KNOW it has similar/better performance characteristics than the built-in one.  And it doesn't, unless you paid a TON for it ($300+ from LSI or something similar).  Even the modern Marvel SATA 6G controllers fall down with high queue depths, and unless you have a very fast SSD, you'd do better with a "slower" Intel SATA 3G controller.

And the Hitachi 5400rpm Deathstars are fine for media storage and whatnot, but you do NOT want to boot to one of them.  Especially with only 4GB of RAM.  You're not extremely RAM limited with that much, but you still might be swapping from time to time.  Swap file + 5400rpm drive + potentially flaky controller = PAIN.

I still think AV is more likely to be the culprit, but you could be hitting multiple problems.

[Matt]

If you use the modes 'Rename' (the default, which handles moving files) or 'Update database...', all processing should occur in the background.

If you use a mode that copies files, this will show a shell copy dialog (like Explorer) and the program will be blocked.

[JustinChase]

Ooh... Is one of those the system drive?
On the third-party RAID card?

No, unRAID runs from a 4GB flash drive plugged into the USB on the back of that box.  All of the drives are for media.  The unRAID OS only takes less then 149MB, and that includes all the logs and plugins I use.  It's very minimal.

My Desktop machine runs on a 60Gb SSD, with 8GB RAM, running win7 x64, AMD quad core.  It runs fine.

I've just noticed that the file I moved is in the new location, and seems to be complete, but MC is still hung up.  While shutting down everything else, MC came into focus just as I clicked the x to close Firefox, so I actually closed MC, but it's still showing in task manager, but it's memory usage is changing, so something is going on still.

If it's not shutdown when I get done with my other stuff, I'll just restart the whole machine, uninstall Avast, and try moving it back to the V: drive, where it belongs.

[JustinChase]

If you use the modes 'Rename' (the default, which handles moving files) or 'Update database...', all processing should occur in the background.

If you use a mode that copies files, this will show a shell copy dialog (like Explorer) and the program will be blocked.

I'm using "Rename, Move & Copy files..." on the 3.3 Gb file located at J:\newsgroups\Canada\Race.mkv and it gets changed to V:\Racing\Formula 1\Season 2012\Formula 1 - s2012e07 -  Race.mkv

[glynor]

My Desktop machine runs on a 60Gb SSD, with 8GB RAM, running win7 x64, AMD quad core.  It runs fine.

This is the machine we cared about the specs on, not the unRAID box.

Are you using the AMD AHCI drivers or the Microsoft ones?

[JustinChase]

This is the machine we cared about the specs on, not the unRAID box.

That wasn't what was requested.

What's the story with the server? It's drives? If the drives are standard home discs (not enterprise class) or even worse - connected to a standard consumer grade motherboard and all are sharing the same bus on the box - the bottleneck could be  happening right on the machine itself. Especially when moving things between drives.

Some of the slowest file moves I have seen here are "interdrive" moves on the same box.

Let us know what the server has for horsepower...

VP

I don't know what my Desktop machine really has to do with moving files from the server to the server, since the file is NEVER on my Desktop.

Anyway, it just came up and asked me if I wanted to remove the empty folder; 1 hour and 36 minutes after I requested to move a 3.3GB file.

I'm not sure which drivers I'm using on the desktop, the AMD or the Windows drivers, but if it still matters, and you can tell me where to look, I'll happily let you know.

Also, I do have it set to output to a log file, but I'm not sure where to find the relevant file to provide.

[Vocalpoint]

Moving from one share to another share has now taken 1 hour exactly, and MC is still not responsive.

Seems to me - this is physical drive contention issue if this is taking an hour.....

VP

[JustinChase]

Seems to me - this is physical drive contention issue if this is taking an hour.....

VP

Maybe, but if I move it to the same share (which can, and usually does actually move to a new physical drive), and it only takes 13 seconds, then how is that possible.

Like I said, it actually moved the file long before MC responded to me again.  I don't think it's actually taking that long to move the actual file, just for MC to "finish".

[glynor]

I don't know what my Desktop machine really has to do with moving files from the server to the server, since the file is NEVER on my Desktop.

Anyway, it just came up and asked me if I wanted to remove the empty folder; 1 hour and 36 minutes after I requested to move a 3.3GB file.

It does.  It certainly impacts the performance of the system, which is why I asked.  Perhaps that isn't what VocalPoint was asking, but I wanted to know about the machine that MC is running on.

When you move a file on a remote server from volume-to-volume from Windows, the machine initiating the move reads the file onto the local machine and then the local machine writes it back out (it "streams" the read/writes, but it is still the local machine doing it, not the remote server).  In this case, network performance matters a LOT.  Disk performance (including the local disk) matters.  The local machine, and network, can often be the bottleneck.

When you rename a file (within the same logical volume), almost nothing on disk changes.  In this case, all the local machine has to do is update the directory entry for the file, which is essentially a "tagging change" in the filesystem.  This is a very small change, and should happen instantaneously, regardless of the size of the file.  For example, on my RAID at the office here, it doesn't matter if I rename a file that is 900GB or 900MB.  They both happen instantly.

Sooo.... Why is your within-the-volume rename (which again, is akin to writing a single line in a text file) taking 13 seconds to complete?  That's WAY too long.  You're calling that "good" but I'd call that horrific.

But either way...

If MC is being blocked, then something is happening on the PC where MC is running.  Like I said (and Matt confirmed), there shouldn't be an issue otherwise, as the Rename is being done in a separate thread.  Now, it is possible that something with this bug is killing your performance, but I don't think so.  I still think the problem with MC being blocked is probably due to Anti-Virus.

But... Wow.  1.5 hours to move a 3.3GB file?  That file-copy performance is terrible.  What kind of throughput are you getting on those volumes?  That seems worse than the Drobo's I've tested, and they're bad.  Maybe the problem is just that your disk is so absurdly slow that it is overwhelming even the limited background updating that MC does in the foreground.  I'm still skeptical, but... Wow.  That's bad.

[glynor]

I can look up the AHCI stuff when I get home.  I don't have any AMD systems here at the office to look at how it shows in the Device Manager.

You want the Driver Provider in the Device Manager for the ATA Storage Controller (or whatever it is called on your system) to be listed as Microsoft.  Not AMD or ASUS or Dell or whomever makes your PC.

[JustinChase]

You want the Driver Provider in the Device Manager for the ATA Storage Controller (or whatever it is called on your system) to be listed as Microsoft.

It is Microsoft.

but... Wow.  That's bad.

I completely agree.  How to resolve is still the question.

[JustinChase]

Okay.  I restarted my Desktop system, and cut/pasted the file from J: to V: with windows explorer, and it estimates it will take an hour.  Of course, this is terrible, but it's also because I'm connected via wifi, and as Glynor pointed out, it's actually copying from the Server, to the Desktop, and back so not terribly surprising that it would take this long to copy 3.3Gb over wifi.  (why must it copy the file to the requesting machine, then back; it seems awfully wasteful?)

However, this just takes me back to my original post, why would MC be unresponsive while it's moving.  Even though it takes way too long, due to wifi, I would expect MC to just let it happen in the background, and MC should respond to me while it's happening.  It sounds like Glynor experiences this, but I don't.  How can i resolve this?

I'm still happy to provide any logs, if they would be helpful, and if someone tells me where to grab them from.

Thanks again for all the help.

[Vocalpoint]

It's actually copying from the Server, to the Desktop, and back so not terribly surprising that it would take this long to copy 3.3Gb over wifi.  (why must it copy the file to the requesting machine, then back; it seems awfully wasteful?)

This makes no sense at all. Your desktop machine has nothing to do with copying a file between shares on another box ?

Something else must be up here...and yes - over wi-fi - any huge file will take forever to copy.....

VP

[JustinChase]

This makes no sense at all. Your desktop machine has nothing to do with copying a file between shares on another box ?

Something else must be up here...and yes - over wi-fi - any huge file will take forever to copy.....

VP

You seem to contradict yourself here.  If my desktop machine has nothing to do with copying between shares, why would it's connection speed to the shares matter?

[Matt]

However, this just takes me back to my original post, why would MC be unresponsive while it's moving.

I couldn't reproduce this.  Anyone else?

Is there some action in MC that causes it to become unresponsive?  If you try to close while it's working, it will show a wait message.

[Vocalpoint]

You seem to contradict yourself here.  If my desktop machine has nothing to do with copying between shares, why would it's connection speed to the shares matter?

When I mentioned connection speed -I meant that "in general" copying files wireless takes a lot of time. But you are saying that this is supposed to be an "intra-share" copy on the server box itself.

So if you are opening Explorer on your desktop machine, then pulling up the shares on the server and copying a file from one share to another - on the server - I can't understand how/why you would say this: "It's actually copying from the Server, to the Desktop, and back" Or how you would know that the file is taking this path.

But - if it is somehow going to your desktop first AND then back to the server - via wi-fi - it's gonna take forever.

Now - when I open the Music share on my server - from my desktop machine - and select a file from Music and copy it over to Public (on the same server) - my desktop machine has no role/connection or purpose whatsoever in the actual copy process. The file copies from one share to the other on the server...all I do is watch the action from my screen.

I guess I do not "get" how you are determining that the file is coming "back" to your desktop and then then going back to the server...

VP

[JustinChase]

I couldn't reproduce this.  Anyone else?

Is there some action in MC that causes it to become unresponsive?  If you try to close while it's working, it will show a wait message.

No specific action, other than the "Rename, Move & Copy..." action that initiates the move.  MC is fine until I hit Okay, then it barely responds to me.

I mentioned earlier that I accidentally clicked the x to close MC, and MC closed, but stayed in the task manager, using 125,xxx K of memory under Processes tab, but was not visible in the Applications tab, and appeared closed by the icon on my quick launch taskbar; but no warning message was presented to me; until about 30 minutes later when the "Do you want to remove the empty folder" dialog box was presented.  When I hit yes, it closed, and nothing further was presented.

Will logs help you find the cause?

[struct]

i too have mc become unresponsive on a copy.  i wanted to use mc to back up music files using rename.move.copy but that means i have no mc for half a day.  it appears like normal windows dialog and mc won't respond till it is done.  using w7.latestmc.no virus.most other things vanilla.

[edit: my bad, didn't read thread carefully, copy is known to hold focus, see Matt's posts above and below.  pity, though??]

[JustinChase]

When I mentioned connection speed -I meant that "in general" copying files wireless takes a lot of time. But you are saying that this is supposed to be an "intra-share" copy on the server box itself.

So if you are opening Explorer on your desktop machine, then pulling up the shares on the server and copying a file from one share to another - on the server - I can't understand how/why you would say this: "It's actually copying from the Server, to the Desktop, and back" Or how you would know that the file is taking this path.

I was simply restating what Glynor said...

When you move a file on a remote server from volume-to-volume from Windows, the machine initiating the move reads the file onto the local machine and then the local machine writes it back out (it "streams" the read/writes, but it is still the local machine doing it, not the remote server).  In this case, network performance matters a LOT.  Disk performance (including the local disk) matters.  The local machine, and network, can often be the bottleneck.

[JustinChase]

i too have mc become unresponsive on a copy.  i wanted to use mc to back up music files using rename.move.copy but that means i have no mc for half a day.  it appears like normal windows dialog and mc won't respond till it is done.  using w7.latestmc.no virus.most other things vanilla.

I'm so sorry for you; but so very happy that it's not just me   Thanks for chiming in!

[Vocalpoint]

I was simply restating what Glynor said...

Hmmm...that is news to me. I am gonna have to check this out later...

VP

[Matt]

i too have mc become unresponsive on a copy.

Like I said above, copy will show a normal Explorer file copy dialog and block the program.  This is expected.

[Alex B]

Glynor is right, when the desktop PC is used for moving files between two regular network shares it reads data from one share and writes it to another share simultaneously (in small packets) and the available network bandwidth will get maxed out. The remote desktop cannot directly instruct the server OS to perform the task on the server. Of course this is wasteful and it would be nice if the operating systems could handle the operation without unnecessarily moving data through the LAN.

Justin, what happens if you use Windows Explorer for moving a huge file from one share to another? Is it faster and can you use MC for other tasks during the move operation?

[JimH]

I'm going to move this to the Hardware board for now.  If it turns out to be an MC problem, I'll move it back.

[JustinChase]

Justin, what happens if you use Windows Explorer for moving a huge file from one share to another? Is it faster and can you use MC for other tasks during the move operation?

Interesting.  I had shut down MC a while ago.  I started the move from the J: drive to the V: drive using windows explorer a while ago, and it's still copying (very slowly).

I clicked to open MC, and it's been almost a minute so far, and MC won't even open.  It shows in task manager as a process, with the Memory usage slowly climbing, but doesn't show in Applications tab, and doesn't show as open on the icon in the quick launch bar; which tells me that perhaps it's just the network bandwith being maxed out that is preventing MC from responding.

It's a couple minutes later, and MC still hasn't opened.  This is from an SSD on the Desptop machine, which normally takes just a second or 2 to open MC; but the library is on the server.

I can navigate to the library using Windows Explorer.  It goes slow, but will display the server location in Explorer.  It's also being used by the HTPC to play music to Patio zone right now; without issue (the same library, no clients).

Probably 5-6 minutes since I clicked to open MC, 20,000 K Memory usage so far.  Wait, it JUST NOW opened MC; but is only showing a black screen at this point.

I think you're onto something.  It must be network saturation that is causing me issues.

Any ideas on how to work around this until i can run an Ethernet cable here?  It's almost 100 degrees every day now, and my attic will reach upwards of 140 degrees, so I'm not terribly excited to crawl around up there in the insulation covered from head to toe right now

A couple minutes more since MC "opened" and it's still just a black screen.

[Alex B]

Could you simply change to use a single share? I think this should be easily possible with UnRAID.

Then any move task that is initiated on the remote PC would technically be a simple renaming task (i.e. no data would be moved between the two shares) and the server would handle the task as it sees best.

[glynor]

Is your Library stored on the unRAID?

[JustinChase]

Could you simply change to use a single share? I think this should be easily possible with UnRAID.

Then any move task that is initiated on the remote PC would technically be a simple renaming task (i.e. no data would be moved between the two shares) and the server would handle the task as it sees best.

I suppose this could work, but it's preferable to me to have the different media types separated, and different drive letters is the 'best' solution to that I've come up with so far.  Now that I know this is an issue, perhaps I'll need to re-think this.

Is your Library stored on the unRAID?

Yes, at J:\MC\libraries\MC17Server

[glynor]

Yes, at J:\MC\libraries\MC17Server

Then that's why MC is becoming non-responsive.  The Library must be on a fast disk.  It doesn't need to be local, but it has to have reasonable read/write and random access rates.  Everything that MC does, from loading to playing a file to file management tasks, accesses the Library (and typically both reads and writes to it).

You are, effectively, running MC with your Library on a bad USB1 thumb drive (only a hop-skip-and-a-jump faster than a floppy drive).  And, worse, when you do a rename operation, you are trying to both read/write the Library itself and to the data store on the volume simultaneously.  So that drive is hopping around seeking back and forth.

1 hour and 36 minutes to move a 3.3GB file means you are getting around 4.8mbps (bits, not bytes) on a sustained, sequential read/write operation, and random access times are almost certain to be far, far worse.  For reference, USB1 has a data rate of 12mbps (real-world speeds are often around 8mbps), so you're running at around 1/2 USB1 speeds on that share.  A WD Scorpio Blue (a run-of-the-mill 5400rpm laptop drive, which are worse than 3.5" desktop drives) will average around 580-590mbps for reads and writes (73-74MB/s).  You're getting less than 5mps (0.6MB/s).  And, I bet your random access times are worse.

You cannot have your library on those volumes until you fix your throughput problems.  With rates that slow, either something is seriously screwed up in the unRAID or (more likely) you have serious network problems.

Get yourself a copy of HDTune and figure it out.  But for now, move the Library to the local disk.  You have no other choice.

[Matt]

For anyone reading along, Glynor gave great advice.

I would recommend keeping your library (and thumbnails, etc.) on the fastest disk you have.  A local drive connected with SATA (and preferably an SSD) is best.

[JustinChase]

I won't disagree that MC should be on a fast drive. 

However, my library is on a SATA drive on the server, not on a thumb drive.

In this case, that drive is only available to me via wifi, which is the cause of the slowness I think; not the drive speed.

With all that said.  How exactly will I use the same library on the Desktop machine AND the HTPC, and the other machines in the house without having it on the Server?

I can have a local copy on the Desktop, which will make that one machine super fast, but will also require me to have the Desktop turned on for the other machines to use the library, and the could only connect via wifi, so I'm not really better off.

I could put it on the HTPC, which also has an SSD, but again, that machine will need to be on for any other machine to use MC, and I'll only be able to connect to it via wifi from the Desktop, so it might not help me much with the original problem here.

I do NOT want multiple libraries.  It makes no sense for me to have to maintain multiple libraries.  I want to be able to change/update information ONE TIME, and allow all MC connections to benefit from that work.

So, what is the solution?

[Matt]

One server machine with Media Center running in Library Server mode is best.  Then connect from Library Server clients.

When Media Center serves the library, the client gets the library and runs it off its local disk.

This is a lot like the SVN / Git model, and I think it's perfect.  Full speed usage of a remote resource, with only changes sent across the wire.

[JustinChase]

Thanks again Matt. 

What if my server is running Linux?  I don't know of any way to install a windows program on unRAID (although it may be possible with some of the various plugins), so I don't currently have any way to have MC running on the only machine that I keep on 24/7.

What's a good plan B in this case?

I guess I'll have to move the library to the HTPC (which is at least connected to the server via Ethernet) and have any other machines force it awake when they try to connect, which I think I've read works pretty reliably.

Any other ideas?

[Matt]

What if my server is running Linux?  I don't know of any way to install a windows program on unRAID (although it may be possible with some of the various plugins), so I don't currently have any way to have MC running on the only machine that I keep on 24/7.

Library Server requires Windows.

What's a good plan B in this case?

Windows in a bottle (VirtualBox).

[glynor]

However, my library is on a SATA drive on the server, not on a thumb drive.

I understand that.  You're right.  Your situation is actually worse than if it was on a USB drive.  Your server is performing at 1/2 the speed of an old, crappy USB1 flash drive.  It is performing at around 1/100th of the speed of your average crappy 5400rpm laptop drive.  You just can't push a melon through that hose.

That is the root of the problem.  You can't have your Library on that server with those performance characteristics.  Your choices are:

1. Move the Library to a volume that doesn't perform like crap.
2. Fix your server's performance problem (whatever it is).

I agree, the problem with #2 is likely that you are running on the edge of viable WiFi range.  Or your router is crapping out.  But it could be a combination of issues.  unRAID is not fast either, all on its own.

[glynor]

For now, it depends on your goals, but the easiest thing to do would be:

1. Move the Library to a local drive on one of your machines.  Preferably one with a good network connection.
2. Share that Library out via the Library Server function of MC.
3. Connect to that shared library from other copies of MC.

This does mean that whatever machine is running the Library Server needs to be left on all the time.  But you have no other choice other than to fix the performance of your server.

But, to explain, the problem isn't just that you have the Library on a network drive.  As Matt indicated, it is BEST to have the Library on the fastest disk you can scrounge up.  But you don't have to.  I regularly connect to my Library at home directly over the network, even WiFi, using my Laptop.  It doesn't perform as well as the systems using the Library directly, or as well as using MC's built-in Library Server functionality, but it is acceptable.  But my throughput over WiFi is 1000x better than yours is getting.

Seriously.  Probably more than 1000x.  The problem isn't that "conceptually" putting the Library on a network drive is bad, the problem is that your network drive is so terrible that it can't be used that way.

[JustinChase]

Your situation is actually worse than if it was on a USB drive.  Your server is performing at 1/2 the speed of an old, crappy USB1 flash drive.

I'm not disagreeing; but I'm curious how you came to this conclusion.

unRAID is installed from the thumb drive (USB2 I believe) when the server starts, then runs from memory the entire time it's running.  Very little, if anything actually runs from or uses the thumb drive on the server, other than initial startup.

The library server is sitting on a SATA drive, so I'm not sure why you think it's performance would be 1/2 of a USB1 connection.

I can stream 1080p and I think 1080i from the server without issue, so it seems to be plenty fast enough.

Again, more curiosity than anything.

Thanks again for the help.

[glynor]

I'm not disagreeing; but I'm curious how you came to this conclusion.

Math.

A file copy is the easiest thing for a drive to do.  That's all sequential reads (assuming your drive isn't terribly fragmented), and sequential writes.  That's a best-case scenario for a spinning drive.

3.3GB = 3379.2MB
1 hour 36 minutes = 5760 seconds

3379.2/5760 = 0.58666MB/S = 4.6933mbps

So, if you're getting 4.7mbps throughput on a simple file copy, then you're probably getting 1/2 that for random access read/writes (maybe worse if the problems are due to network errors, so lots of errors and resends are needed for lots of "small changes").