Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[MagnaCartaHG]

I lost a bunch of files due to a hard drive failure. I ran some data recovery tools and was able to recover quite a bit of the data.

Before the data recovery software ran, CHKDSK moved the files to new hidden directories on the same drive. MC Auto-import was set to - Yes (protect files on missing drives), and the drive was connected so it assumed that the files had been deleted.

- I have backups, but I waited probably about month to get a new hard drive and restore them so the backups will remove all my new files that I've updated with custom metadata. Is there a way to roll my database state back a month and somehow merge the new data into the database without it overwriting existing data?

- I also have an old copy of MC21 running on the machine that wasn't running when the data was lost, so it behaves like a server where the drive was disconnected and still has all the data. I was able to export the files to an MPL file, but my new MC instance doesn't let me import an MPL full of broken links. Is there a way to force it to accept the files from an MPL file or have it import data from another server that's running?

It would be most magical if I could preserve the old thumbnails (some of the files are videos) and I can think of some clever tricks to export all the data from the old drive using MCWS, but wondering what experts would recommend. In the end, I'm content with running my old MC21 library if I want to see the old files, but I really want 1 consolidated library with the original import dates and everything.

Thanks in advance, and by the way, thanks for this amazing piece of software. It's one of my favorite things to buy every year

[JimH]

It would probably work best if you moved your files to the locations where they were originally.  If possible.

Then restore a backup in MC.  MC makes them automatically and dates them.

[MagnaCartaHG]

Thanks. CHKDSK preserved many but not all file names so I think MC would detect some as moves and the rest as new files with names like "File001"  and still delete data for files that were moved to unrecognizable locations. (there might not be too many files like this).

The bigger problem is that I've added a lot of custom data to my library after the fact so I'd like to somehow retroactively add to the current library.

I did a quick test, and I'm able to import an MPL where I've modified Filename for 1 lost file to point to a new path and it's able to make the playlist which contains the 1 file I modified with correctly preserved date imported, play counts, and everything else. I may just write a script to either modify the playlist to the restored file paths or move the restored files and call it a day.

* Right now the files show up in "Imported Playlists", but they're not in my library. Is there a way to make them appear in the library or at least appear in search? The drive is not being watched by auto-import, if that makes a difference. I tried making a View Scheme with only the files imported playlist, but nothing shows up.

[wer]

I was able to export the files to an MPL file, but my new MC instance doesn't let me import an MPL full of broken links. Is there a way to force it to accept the files from an MPL file or have it import data from another server that's running?

Yes, in a way.

An MPL playlist is a plain text file.  Edit it in a text editor, and use search and replace to change the paths to the correct values for your "new" copy of MC. 

When you import an MPL playlist, IF the path for the playlist file matches the path for a file in the library, then MC will update the library with the tags from the MPL, with one exception: it will not delete values.  If a tag like "Last played" has a value in the library, it will NOT overwrite it with a null value from the playlist.

[MagnaCartaHG]

@wer Thanks. I think editing the XML is going to be the path forward for me. The filenames are unique, so it should be easy enough to update the MPL to have the new locations. I can use the length of the imported playlist as a proxy for if all my files paths were restored, and manually go through the files with unrecognizable names.

Do you happen to know why files imported from a playlist don't show up in the library, but do in the playlist tree? Is it because the path is not part of auto-import list? Assuming M: for old path and N: for new path, should I do something like:

* Fix M_Drive.mpl so that all the files match
* Import the playlist
* Add N: to the auto-import paths
* Run auto-import

Will MC "remember" that its seen these files in a playlist and not overwrite existing data when bringing them into the main library?

[glynor]

Instead of editing the MPL files in a text editor one by one, which is a nightmare (unless there is only a handful) what about:

0. Make a Library Backup if you don't have one.
1. Use RMCF in Update Database Only mode to "move" the files to their new locations (thereby "fixing" the links).
2. Export your MPLs now. The file paths will be fixed.
3. Use those after you restore to update the files needed.

[wer]

Instead of editing the MPL files in a text editor one by one, which is a nightmare (unless there is only a handful) what about:

It's not about the number of them, it's about the complexity.  Remember, I said search and replace. Only a ned would try to do them one by one.

If your path change is more complicated than can be done with a simple global search and replace,  then glynor's idea is better.

There would be an additional step 4 though:
4. In the old library, Restore your library backup from before the RMCF to undo the fake RMCF you just did. (Or just redo the RMCF backwards)

Magna, to your question, importing a playlist doesn't actually import any tracks. It imports a playlist. A playlist file is an index to some media files, it does not contain the media files themselves.  If you import a playlist and expect it to work, the media files must also be imported. If you expect MC to update tags based on playlist metadata, then the referenced media files must ALREADY be in the library at the time of import. Read carefully what I told you before.

Have the files in your library already. Then import a playlist that refers to those already existing files, and MC will update the tags.  This is what I said before.

[MagnaCartaHG]

@wer - Thanks for the explanation, I misread your original post and it makes sense now.

@glynor - I wasn't planning to edit the XML by hand, but I do need to parse the XML and script with the Fields that are Filename. I have Ruby scripts for reading MPL.

Moving the files to their original location is non-trivial because of the mess CHKDSK has made of my files.

To give a concrete example, it's not that it restored `M:\Movies\HD\Sci-Fi\2001 A Space Odyssey\2001 A Space Odyssey.mp4` to `N:\Restored\Movies\HD\Sci-Fi\2001 A Space Odyssey\2001 A Space Odyssey.mp4` - CHKDSK restored `M:\Movies\HD\Sci-Fi\2001 A Space Odyssey\2001 A Space Odyssey.mp4` to `N:\found.000\dir0103.chk\2001 A Space Odyssey.mp4` with a random assortment of other movies. The only way I know the file should be in "Movies\HD\Sci-Fi\2001 A Space Odyssey" and not `found.000\dir0103.chk` is because of the exported MPL file of everything on the `M:` which I made using the old instance.

To get the files in the right place, I first need to rebuild the old directory tree. I don't see a way other than writing a script that matches on the equivalent of the Filename (Name) in each item of the MPL file, i.e. "2001 A Space Odyssey.mp4", searches the "N:\" to find the new location in "N:\found.000\dir0103.chk", and creates the folders and moves the file on disk to "N:\Movies\HD\Sci-Fi\2001 A Space Odyssey\2001 A Space Odyssey.mp4".

I don't want MC to import `N:\found.000\dir0103.chk\2001 A Space Odyssey.mp4` and if I get what @wer is saying, I need to align the files to the right paths and import them into the library before if I import the MPL or the data won't get updated, or effectively backdated in this case.

Given that the old drive was M:\ in the MPL and I restored them to the N:\ drive. If I:

* Run the script to perform the file moves and rebuild the old file structure
* Find replace M:\ with N:\ in the MPL
* Tell MC to import the N:\ into the library
* Import the modified MPL with old data and new paths

I should end up with:

* All my current data including items created since the data was lost
* All the lost files in the library with the new paths, but data as it was in the MPL exported from the old instance, including the old Date Imported, Play Count, etc.

Is that correct?

[wer]

Yes, that is correct.

However, for the video files, don't you have sidecar files?  The metadata for the video files should have been saved in sidecar files in the same directory as the video files (originally), unless you turned off sidecar files (which is foolish).  If the sidecar file is present when the video file is imported, MC will load the metadata from that.

You will need to update, inside the sidecar file, the path to its media file.

If the sidecar files were not preserved, then you indeed lost that metadata, except for what is in your old library.

For audio files, I hope you were storing metadata in the file tags.  If you weren't, I hope this has taught you that lesson.

Bottom line: I always recommend to people they write their tags to the media files themselves, or sidecar files for types that don't support full tagging (like video).  I recommend that for exactly his situation and this reason: if you had done that, you would have your metadata as long as you had the files, and you wouldn't be forced into trying this stunt with the MPL.

By the way, there is nothing inherently wrong with importing from those chkdsk dirs, as long as they are clean in the sense they contain only what you want to import. You could then use RMCF to move them based on metadata, if you had the metadata in the sidecars or file tags.  I would be more worried that chkdsk "recovered" a corrupt file instead of a good one. I would wish I had a backup.

Clarify two things:
The MC that was running lost the file handles due to auto import. (Which should teach you to be more careful of auto import.) so it has essentially nothing useful.

And that the MC21 that was NOT running still thinks the files are in their correct location on M:, and NOT in those chkdsk directories. Is that right?

Have you looked for the sidecar files?

[RoderickGI]

All good advice above.

I haven't completely visualised all the issues, but on this issue...

Moving the files to their original location is non-trivial because of the mess CHKDSK has made of my files.

To get the files in the right place, I first need to rebuild the old directory tree. I don't see a way other than writing a script that matches on the equivalent of the Filename (Name)...

... I will add another idea.

You can use MC to move the files to their correct locations without first importing them. Just use the MC Explorer (Navigation Bar > Drives & Devices > Explorer) with the top settings of "Only files not in Library" and probably "Show File In Subfolders" to display whats in the Chkdsk directories and then use the RM&CF function to move them to the correct directories, where they came from originally, although on the N: drive not the M: drive. If your file naming is as per your example, then the RM&CF move should be easy. There is only one thing you need specifically, and that is to get the filename without extension to use in the path. To do that you use the Filename() function. i.e.

filename([Filename (name)],0)

Returns the filename without the extension.


So you could just open the "N:\found.000" directory using the MC Explorer showing all files not in the Library, perhaps apply a filter to select specific file types, such as MP4's, then set up a directory rule to move the file to the correct location.

Base Path: N:\Movies
Directory Rule: HD\Sci-Fi\filename([Filename (name)],0)\

Of course, that relies you knowing something about the files, such as the format (HD) and Genre (Sci-Fi), but you would need to set that up in any script anyway. For audio that is easy as the MC Explorer will read tags to display the files. For video, not so easy, particularly if Sidecar files are not in the same directory.

I'm not sure that the above improves the recovery process or just muddies the water. But I thought I would mention the functionality. It could save some complex scripting work. It could also be used to move Sidecar and Cover Art files without importing them, although Sidecars would be a little more tricky.


Of course as Wer says, you could just import the files while they are in the Chkdsk directories, then move them, then use the MPL files to restore tag information.

[MagnaCartaHG]

Base Path: N:\Movies
Directory Rule: HD\Sci-Fi\filename([Filename (name)],0)\

Of course, that relies you knowing something about the files, such as the format (HD) and Genre (Sci-Fi), but you would need to set that up in any script anyway

Thanks for suggestion. As you said, I have to know something about the file, and that's part of the problem. Some schemes can be generated from metadata, but sometimes the folder I choose is my organization scheme even though the embedded data is correct for the scheme created by an external system or publisher. I can probably do better by putting everything in flat content type folders and using views and playlists within JRiver, but I haven't done so. You did give me an idea of how to get JRiver to do the move however, so thanks again.

Clarify two things:
The MC that was running lost the file handles due to auto import. (Which should teach you to be more careful of auto import.) so it has essentially nothing useful.

And that the MC21 that was NOT running still thinks the files are in their correct location on M:, and NOT in those chkdsk directories. Is that right?

Have you looked for the sidecar files?

1. The MC that was running 100% does not have the files. Here's the sequence of events:

* Auto import is watching M:
* M drive fails (this is when I should have stopped watching M:/ or used another computer to do the restore)
* CHKDSK runs, moves most files to found.000, leaves 1 file in its original location, and reboots Windows
* MC starts with Windows and detects all but 1 file has been moved out of directories that it watches
* MC updates the library to have 1 file in M:
* I recover the data to a new drive and disconnect the M:
* MC says the 1 file on the M: is "Not available" but in the library

2. That is correct. MC21 shows all the files. Since I now know that I can update the metadata, I'd rather re-export a copy of the metadata from a more recent backup of my current MC. I just used the old MC21 because it was around and was trying to validate the strategy. I now have a recent enough backup of the data in question from before the data loss.

3. I agree that if I had sidecars this would be a lot easier, but I don't (I have a few, but maybe 10% of the files). I get the advantage - that the library data would have been recovered in the same state as it was lost in.

You will need to update, inside the sidecar file, the path to its media file.

I don't want to update the sidecar to point to the CHKDSK organization scheme, but you make me wonder if I could break up the large MPL into sidecars for each file since a sidecar is just a 1-Item MPL with title JRSidecar.

If I want JRiver to do the file moves, will this work?:

* Find each item in the MPL and write the MPL data to its new N: destination as a 1-item sidecar
  * Find `N:\found.000\dir0103.chk\2001 A Space Odyssey.mp4`
  * Match on the original filename and write a 1-item MPL with the original data to the new location
  * For example `N:\found.000\dir0103.chk\2001 A Space Odyssey_mp4_JRSidecar.xml`
* For each sidecar, Copy the Filename field to a custom field "Original Filename", with drive letter swapped
  * <Field Name="Original Filename">N:\Movies\HD\Sci-Fi\2001 A Space Odyssey\2001 A Space Odyssey.mp4</Field>
* For each sidecar, Update the Filename field to point to the new N: destination
  * <Field Name="Filename">N:\found.000\dir0103.chk\2001 A Space Odyssey.mp4</Field>
* Create a field "Original Filename" in my current library to support the incoming gdata
* Import the files into JRiver from "N:\found.000" with data coming from the new sidecars
* RMCF based on the "Original Filename" field

(I'm guessing the RMCF will fail because it won't let me include slashes from the "Original Filename" field. Worst case I move the files with code)

@wer - can you elaborate on how JRiver manages sidecars? Do I have to move and rename files in JRiver to have them stay up to date? Is there a way to tell JRiver to batch create all MPLs in my library if I turn on the setting now (which I have done)?

if you had done that, you would have your metadata as long as you had the files, and you wouldn't be forced into trying this stunt with the MPL.

That makes sense. The sidecars would act as an external backup and I wouldn't need library backups and old playlists. But either way I'd need to read the sidecars to restore the directory tree. And besides, stuntin is a habit and habits are hard to break.

I would be more worried that chkdsk "recovered" a corrupt file instead of a good one. I would wish I had a backup.

I wouldn't have run CHKDSK if I had backups.

[wer]

Yes, well the full extent of the pickle you're in has become clear.

So with sidecar files, you don't get to manage them. You can turn them on or off. MC always keeps each sidecar file in the same directory with its corresponding media file. If you tell MC to move the media file, it will move the sidecar.  As long as you have sidecars enabled, and if you have "update tags when file info changes" enabled, then MC will automatically write library changes to the sidecar files and keep them up to date.

So if you only currently have sidecars for 10% of your files, let's just put that aside, and focus on getting the data from the intact library; that path forward is clear. Once everything is put to rights, then you can enable sidecar files and "update tags when file info changes" for the future.

I'm not trying to rub your nose in it, but I'll just mention that backups need not be expensive for burdensome, since media files change rarely. One can buy a 8TB external drive for $120, hook it up once every few months, and just to a copy/update. Even a simplistic solution like that would leave you much better off than you are at the moment.

So like I said, I suggest we ignore the sidecars at this point, and focus on the clear path to extracting the data fresh from the intact library.  As long as you can retrieve all the media files to their correct locations (one way or another) that method will work for all files and should present you with no surprises.

[MagnaCartaHG]

with sidecar files, you don't get to manage them. You can turn them on or off. MC always keeps each sidecar file in the same directory with its corresponding media file.

Does this also mean that I should no longer manage my files outside of MC to not break the synchronization of the sidecars with the media file? (perhaps moving files under MC's nose is a bad idea in the first place)

Once everything is put to rights, then you can enable sidecar files and "update tags when file info changes" for the future.

Will this mean that only my new files get sidecars? Is there a way to ensure I have sidecars for all my existing files? If not, would making them manually from splitting up full library MPL work?

[wer]

Does this also mean that I should no longer manage my files outside of MC to not break the synchronization of the sidecars with the media file? (perhaps moving files under MC's nose is a bad idea in the first place)

A sidecar file isn't magic, it's just a file. But it's not a great idea to move member (of the library) files outside of MC anyway. Especially if auto-import is on.  Really don't do that.  But I bet you know that now, don't you?    But it certainly can be done if you know what you're doing. If I had to move 10,000 files, I would do it outside of MC, and use RMCF to update the paths afterwards. This is because RMCF is not 100% reliable when moving massive amounts of data. I've seen it choke when moving terabytes, or tens of thousands of files. Hundreds of files it always does fine.

Will this mean that only my new files get sidecars? Is there a way to ensure I have sidecars for all my existing files? If not, would making them manually from splitting up full library MPL work?

Well not all file types normally get sidecars anyway, just the types that don't support tags well. Refer to the "Sidecar tagging mode" option. For file types that support tags, like FLAC, better to just write the tags directly into the file.

Once you turn on the sidecar options correctly, you can force all the necessary sidecar files to be updated/generated by selecting all the files and doing an "Update tags from library". If "Update tags when file info changes" is set, your in-file tags (like FLAC) will also get updated.

I don't think you ever mentioned if all these missing files of yours were video files, or if you also had regular taggable audio files like FLACs involved in this drama.

[glynor]

This is because RMCF is not 100% reliable when moving massive amounts of data. I've seen it choke when moving terabytes, or tens of thousands of files. Hundreds of files it always does fine.

FWIW, I just used RMCF to move 58k files from two different drives over to a new external media drive for my brother-in-law and it worked fine. Pro-Tip: If you're going to do that, make sure:

1. The directory you're moving into is clear. Otherwise you could end up with a million one-at-a-time Do you want to overwrite? prompts for the stupid auxiliary files (which is annoying, but the way it is) that you have to click through one at a time.

2. I think it might be placebo, but I always turn off Auto-Import manually before doing any big moves. We're talking 10 years ago now, but I've definitely seen MC get confused when Auto-Import is on and a giant slow-running RMCF move is in progress.

I think the latter happens because Auto-Import does periodic "full scans" of watched directories. If one of these "every-two-hours" scans happens while the RMCF is still running, it'll sometimes try to re-import the files that had been "moved" (inside MC) but which are still queued for the tag change and so the actual disk move hasn't happened yet.

And then it gets super-confused and broken.

[wer]

Yeah, well maybe so.  And just the other day someone said they used RMCF to move about 300k files and 5+TB, and it took a day and a half to run but worked.

I don't dispute it can happen.   But it can "not happen" too.  I just wouldn't bet my life on it.  I don't use auto-import, but certainly turning off auto import first is very sound advice.

I suspect you're on the right track with your 3rd paragraph glynor, because anecdotally it seems to be more likely to have problems if there are many very large files, like bluray rips, that can talk a long time to finish copying.