INTERACT FORUM

Windows => Third Party Plug-ins, Programs, and Skins => Topic started by: Mirko on September 15, 2002, 07:25:59 am

Title: [Plugin] Duplicate finder
Post by: Mirko on September 15, 2002, 07:25:59 am
@KingSparte:

I did a new thread. But I'm not able to close the old one. Could you?


Mirko
the check for updates button tells me there is an update but the version is older than the running version.



Yeah I know. I have to update this manually, for now. So I did not do this. I have to do some automation.



besides got source for this? I could use it to check to see if there is new database files on my web site.



I have created a file on my webserver (mjdubfinder.ver) where I store the newest version number. I have to do it a better way.
But I only used the functions provided by MJ. I can send you the source, if you like.



Seen a few more problems dealing with play times of the second dup,



What do you mean play times?


and the summery box needs to be extended down a few lines it seems to be cutting off some info if you have many files but it seems to have installed fine.



Thanks for checking. I will make it bigger. Right now ;-)



I was thinking that maybe if there was a dup that maybe the list should have a [X] check box and check the worst file (the best file to delete).



Hm. I thought about that one. So I did a "dup-quote" (the percentage value on the left side). And if you sort by artist or title, you can see the different songs... But I think the checkbox-idea is good. You mean: Check the file to delete and don't use explizit delete-button up on the "button bar"?



I would like to see a Dup is only found when a match is made on the song name, the artist, and the play time is the same.



At this moment, the dupfinder checks for artist and name. The matchquotes evaluates artist, name, duration, filesize and kbits. So I think you are right for adding duration the the first checkings. I did that.

Thanks a lot for helping me out.

I tried it with MJ9. It's _slow_. I have uploaded the new version to this location.

Mirko
Title: RE:[Plugin] Duplicate finder
Post by: David on September 15, 2002, 02:53:43 pm
Mirko,
I am able to open Dup Finder using the new install that you posted.  However, as soon as I run it I get an "error -4" message.  Each time I click OK another error pops up.  The only way to exit is tro kill the application in Task Manager.

The debug log has only one entry:
1 E:\My Music\Various Artists - Nipper's Greatest Hits - The 20's\Maurice Chevalier - Louise.mp3

Thanks,
David
Title: RE:[Plugin] Duplicate finder
Post by: Mirko on September 15, 2002, 08:23:23 pm
Hi David,

is this an error box with my mailadress in it (mwalter@2via.de)? How many files do you have in the listview in the upper part of the plugin?

Sorry for that unfriendly errorhandling. As you might have read, I don't have too much sparetime to spend on this. And I even can't reproduce the errors. But I have an idea, what may go wrong here (maybe "'" is some sort of special character for the function I'm using; maybe I should even through out that compare-function alltogether...).

Thanks for reporting,
Mirko

P.S. I found out why the plugin seems to be so much slower than before. It's the debugging-file. If you compare a couple of thousand files, like I did for testing purposes, you end with a (many) multi-mb debug.lst-file. But I think this helps finding the error...
Title: RE:[Plugin] Duplicate finder
Post by: zak326 on September 15, 2002, 10:33:31 pm
the same error just came up for me. And it sure is persistent it conmes up everytime I try to do anything/.
Title: RE:[Plugin] Duplicate finder
Post by: Mirko on September 15, 2002, 10:52:21 pm
@zak326: Is the error-msg with my emailadress or without? Could you please send the file "debug.lst" in "c:\" to mwalter@2via.de?

Thanks a lot.

Mirko
Title: RE:[Plugin] Duplicate finder
Post by: zak326 on September 16, 2002, 01:31:11 am
It not onlye came up with your address it came up for every song in the playlist is was analyzing. I finally ad to do a ctrl-alt-del to stop it.It was not a fun experience.

I don't know how to send a bug report other than to say it bugged me a lot.
Title: RE:[Plugin] Duplicate finder
Post by: Mirko on September 16, 2002, 01:37:59 am
@Zak326: Thanks for the info. The main problem is: I cannot reproduce this and have no idea, where the error comes from (but I do now, because you provided the info, that my mailadress is there -> so I can find where I brought in the errorhandler and may find the actual bug).


Edit: I found the error. The new version is uploaded, if anyone wishes to try it.
Title: RE:[Plugin] Duplicate finder
Post by: David on September 16, 2002, 04:28:12 am
Mirko,

The new version works great!

Thanks,
David
Title: RE:[Plugin] Duplicate finder
Post by: David on September 16, 2002, 04:37:37 am
I noticed one tiny spelling error.  The plugin is listed as "DubFinder" in the MJ Plug-in and "DupFinder" within the program itself.

The program is chugging away right now without any errors so far.  Only 7,500 more entries to search.

Thanks,
David
Title: RE:[Plugin] Duplicate finder
Post by: Mirko on September 16, 2002, 05:25:37 am
Thanks for the positive reply. I needed that :-)

There is a new version uploaded, which speeds up searching _very_ much (from about 2hrs for 10.000 files to 15mins I think).
So you maybe you would like to update.

Question: Should the plugin save the results for later usage? I think so. But I guess I have to do little check routine then, which finds deleted files during two sessions of the plugin. Would like to know how you think about it.

Hint: The play-buttons (err... the corresponding "stop") don't work with MJ9 yet. Don't know if it's my fault :-/
Title: RE:[Plugin] Duplicate finder
Post by: KingSparta on September 16, 2002, 06:00:03 am
Mirko

This version seems to work better, and don't have the problems i did with the other.

I Did notice "Lable1" that is your Dup Counter is shown, you might want to set this to "0" on Program Init

Also I guess what the program does is add all the dups to the playing now and the user needs to Delete them his self. since you can't delete them from Playing now without using A Key Combo (i forget what it is) on the message that pops up you might want to let them know what the key combo is.

If you are going to allow a check of the play time i was thinking maybe another nice option would be to allow the user to set the seconds it can be close to.

so if Song1 was 3:30 seconds and song 2 was 3:30 that would be a match. and allow the user to set the seconds off it could be match so if he set it to 1 second the results would match if it was 3:29, 3:30 and 3:31.

the other thing to look at would be what is the KBPS if they matched the best out of the two might be the one with the higher KBPS and if the list had a check box. the program could automaticly select the lower KBPS file and place it in the "To Be Deleted" list.

Looking Good However...
Title: RE:[Plugin] Duplicate finder
Post by: Mirko on September 16, 2002, 07:52:41 am

I Did notice "Lable1" that is your Dup Counter is shown, you might want to set this to "0" on Program Init


That's a debugging label and I forgot hiding it. It shows me the status of my duplicatication routine (I had many errors with too many matches and so I can cancel it prior to memory problems).
I will add a mechanism, so that only my username might see this.



Also I guess what the program does is add all the dups to the playing now and the user needs to Delete them his self. since you can't delete them from Playing now without using A Key Combo (i forget what it is) on the message that pops up you might want to let them know what the key combo is.



Actually, the program creates a playlist called "Duplicates". But you are not able to delete out of this, either. So I guess, some time in the future I will add a real deletion routine. But not in this early development stage.



If you are going to allow a check of the play time i was thinking maybe another nice option would be to allow the user to set the seconds it can be close to.



I had an internal quote of about 1.015 to match two durations (and filesizes). I changed it to 2sec by default and you might edit the value in the setup-tab.



the other thing to look at would be what is the KBPS if they matched the best out of the two might be the one with the higher KBPS and if the list had a check box. the program could automaticly select the lower KBPS file and place it in the "To Be Deleted" list.



I show both songs in one row. So what would a checkbox mean? Delete 1st or 2nd? I'm not sure how to do this.



Looking Good However...



Thanks for the help.
Title: RE:[Plugin] Duplicate finder
Post by: nila on September 16, 2002, 08:26:58 am
Wahey!!
This works great for me now!

Here's the thing for me. I dont want to delete any duplicates if they're on seperate albums, only if I have the second copy in my collection of random songs which to make life easier for myself all have their album set to: Random Songs.
This is filled with surprise surprise, random songs that I like at the time and so get.

For me it'd help a LOT if I could filter the results of the duplicates shown so that it only showed duplicates where one of the album names was: Random Songs.

Any chance of any filter like this? :)
Title: RE:[Plugin] Duplicate finder
Post by: Mirko on September 16, 2002, 09:29:03 am
@Nila:

I added an option for "only show duplicate, where one of them has album=xyz". I named it "Nila special option" ;-) You can find it in the new version already uploaded.

Please provide feedback, I can't test it with my DB (and I don't intend to make such a thing *g*).

I also did a smaller update-packages (downloadable here), where only the OCX-file is included. This should be sufficient, if you already had the bigger one downloaded.

@KingSparte:

the other thing to look at would be what is the KBPS if they matched the best out of the two might be the one with the higher KBPS and if the list had a check box. the program could automaticly select the lower KBPS file and place it in the "To Be Deleted" list.


In my archive, there are almost all files encoded using VBR. So it makes no sense with my files to check for KBits, because MJ seems to store the kbits from the first frame (I don't know, if it is the "cut through" (??) then it would be usefull).
Edit: I added this option. Me is not everyone, right?

I made the auto-deletion-options visible.
Title: RE:[Plugin] Duplicate finder
Post by: KingSparta on September 16, 2002, 12:16:32 pm
Mirko

about the check box list

since one would be 320kbps and the other 128 kbps the program should automaticly check it, meaning it is a dup that matched the same play time as file two. the user could then put all the checked songs in the delete list By [Put All Checked In The Delete Box]

I Am More Thinking Of Somthing Like

[_] The Beatles - Help 320KBPS
[X] The Beatles - Help 128KBPS
Title: RE:[Plugin] Duplicate finder
Post by: Mirko on September 16, 2002, 12:46:41 pm
@KingSparta:

Sorry, but I don't get it. You would like to have a checkbox, that "marks" the files matching a specific criteria? So you might find 500 duplicates and would like to check the ones with same duration but different kbits.

Hm. The checkboxes take quite much space in there (and there is not enough for that, is it?). Would you like to have an option (maybe try right clicking on the delete-buttons, you might find something similar) to pre-select some items matchin specific criteria? I could add something like this quite easy. It would make the same you would do by hand clicking on the listitems, selection for deletion.

That's a quite good idea, I think. No automatic deletion during processing, but after that for specific criterias. I like that. I have done that. Open issues: The delete-operations should only work on that selected files. I will add a function to this tomorrow.

Thanks a lot,
Mirko
Title: RE:[Plugin] Duplicate finder
Post by: KingSparta on September 16, 2002, 01:42:58 pm
>> Would you like to have an option (maybe try right clicking
>> on the delete-buttons, you might find something similar)
I just fingered that out, that works good too. :-)

by the way

[Shift & The Delete Key] Will Delete the marked Files In the Duplicates Play List. what i was thinking if a user Pushes "Delete" you have a message box comeup and it tells the users where the files are. you can also tell the user to mark the files and use Shift and push the delete key to delete the files in the duplicates folder.
Title: RE:[Plugin] Duplicate finder
Post by: KingSparta on September 16, 2002, 02:42:53 pm
Playing some more with it.

there seems no way to automaticly select files that are the same.

if the Kbps the size and the play times are all the same. the program will find it as a dup, but the user must then manualy delete one or the other.

I would prefer not to manualy delete and allow the program to select one if everything is the same.

I have 500|PLS| files the size and the play time is not the same on all of them but the program will not select them for deletion. it seems the files sizes are close to the other, and also the playtime but not the same.

Also

if you change this line in the registry.reg file

"PluginMode"=dword:00000000

to

"PluginMode"=dword:00000001

the plugin will stay resident, so when the user goes to the Duplicate folder and then back to your plugin the dup list will still be there, this will allow the user to continue where he left off and there would be no need to rescan the files.

if you do this you will need to make sure when the user presses "Delete" that all files sent to the dups folder is fromved from list 1 (the dup list)

by the way, I would like to know how many dups are in the dup list do do not hide it.
Title: RE:[Plugin] Duplicate finder
Post by: Charlemagne 8 on September 16, 2002, 04:22:26 pm
You would have to be able to globally set preferences for automatic selection of the desired file to delete. There actually could be someone that WANTS to delete the 320 KB instead of the 128 KB file.
Title: RE:[Plugin] Duplicate finder
Post by: Mirko on September 16, 2002, 08:25:13 pm
@KingSparta:


there seems no way to automaticly select files that are the same.


There will be this evening (that will be about 12hrs from now).


by the way, I would like to know how many dups are in the dup list do do not hide it.



You mean I should show the "label1"-thing to all users? I thought I did that in the "summary box"... ok, I will add it (without the "randomly set" label).


I have 500|PLS| files the size and the play time is not the same on all of them but the program will not select them for deletion. it seems the files sizes are close to the other, and also the playtime but not the same


"It's not a bug, it's a feature". This will occur, if the difference in duration is not big enough (maybe you didn't change the value on the setup-page?). Is this the case here?

Btw: This resident-thing: I did add a "load" and "save" button to the plugin to store the search results during two sessions. I didn't know that I can change the default handling (thanks a lot for this one!). I think I will change this (and add a button to clean all memory, just in case someone doesn't want to have a 20MB heap allocated all the time *g*).


if you do this you will need to make sure when the user presses "Delete" that all files sent to the dups folder is fromved from list 1 (the dup list)


I don't think so. You can't delete a file a second time, if it is marked as "for deletion". Did you try that? It _should_ be in there.
But for viewing pleasure, it might be a good option.


You would have to be able to globally set preferences for automatic selection of the desired file to delete. There actually could be someone that WANTS to delete the 320 KB instead of the 128 KB file.


I thought about exactly that yesterday and also thought, that nobody would actually want that :-) Ok, I will make it an option.
Title: RE:[Plugin] Duplicate finder
Post by: nila on September 17, 2002, 12:47:37 am
Yes yes yes!!

Your an angel or a god or something :)
That works perfectly.
I love the little name for me too even though the name might seem a little confusing to other users who just discover the plugin.
I dont need it personally because of the fact I name all the albums a certain way (ie Random Songs) but it might be worth adding another option whereby they can either specify an album that one of them needs to match as in my case, or the other choice might be to let users select a specific directory that the song needs to be in as alot of users might keep all their single songs seperate from their albums.

Thanks for my option though, I love it, it's soo pretty :)

When your adding the rules to allow automatic selection of which song to delete, I'm always going to want to delete the one that's in 'random songs'. If you included the path option too that I mentioned above maybe you could include a rule in the 'auto delete' option that allowed the one that matched the album name or the path dir to be the one automatically deleted.

Your plugin is a god send :)
Title: RE:[Plugin] Duplicate finder
Post by: Mirko on September 17, 2002, 01:26:55 am
Update:

I developed a "little" speedup of the search-routine. I have to test it some more (it works with about 9.999 random text strings taking about 5secs), but I guess I can speed up the whole thing a lot. Yesterday the plugin took about an hour for my collection. That's too much for me.
I may do a new alpha version for this testing. If someone is interested in this, I would really like some volunteers. But I have to say: It may not function properly, but it doesn't make any weird things with your database. I will post the link here, when I have included it. This will be in about 8 or 9 hours, because I'm currently at work and have to do something ;-)

@Nila: Thanks for the positive feedback!

Mirko
Title: RE:[Plugin] Duplicate finder
Post by: nila on September 17, 2002, 03:36:48 am
np,
and I'll test it for you. I have around 7000 mp3's :)
Title: RE:[Plugin] Duplicate finder
Post by: KingSparta on September 17, 2002, 03:44:47 am
Mirko

It might helped also if I would have checked for an update

> You mean I should show the "label1"-thing to all users? I thought
> I did that in the "summary box"... ok, I will add it (without
> the "randomly set" label).
yes but it does not tell you as the process is working.

Also if you do another search, the summery for the last search does not clear. if you could make the summery add files as it finds dups that would be good. I am not sure if this would slow the search down allot.

I still do not see how to delete the same exact file. if you take a file and make a copy of that file and load it into MJ that is the one i want to delete, not the one off 5 seconds.

Since the program only will select it to be deleted if it is off so many seconds how about revirsing that so if it is off so many seconds do not select it for deletion.

[X] Reverse Delete Selection

This will select all files under the Default "2 Seconds" but leave the other files that are over 2 seconds or what ever a user selects.
Title: RE:[Plugin] Duplicate finder
Post by: Mirko on September 17, 2002, 04:20:39 am
@KingSparta:

Is the file found as a duplicate? Did you order the listview by matchquote ("%") by clicking on that column?
There seems to be a missing option for automatic selecting same files. But I'm curious, if it is found...

Thanks,
Mirko
Title: RE:[Plugin] Duplicate finder
Post by: KingSparta on September 17, 2002, 05:33:14 am
>> Is the file found as a duplicate?
YES, if you have two files and one is a copy of the other it finds it as a dup, there is no way of deleting it unless you do it manual.

% is 058% on all dups like this, all fields are the same, Size, Bytes And Kbps

-----------------------------------------------------------

if i am not mistaken if you set the seconds off to lets say 2 it must be off 0-2 seconds to match? or More Than 2 Seconds To Match?

If i was to set this to 2 seconds i would think from 0 to 2 seconds the program should find a match and put it in the auto delete list.

-------------------------------------------------------------------

I would like to delete all files that are exact matches, give or take 2 seconds (anything over my setting, it may be another version of the same song), also the one with the lower KPBS, also the one with a smaller file size.
Title: RE:[Plugin] Duplicate finder
Post by: Mirko on September 17, 2002, 06:48:12 am
@KingSparta:

You should get a "100%"-quote for exact matching files. I checked it with my database (I did copy some files for testing this) and they all got 100%. And if you get 100%, you can auto-select them using the right mouse button on the delete-button.

I also did the tuning you suggested with the reg-file. Now it stays ;-)

@Nila, @all:

I included the new routine for searching duplicates. It works quite good, so good in fact, that I integrated it in the "normal" version.
Title: RE:[Plugin] Duplicate finder
Post by: KingSparta on September 17, 2002, 08:56:03 am
>> You should get a "100%"-quote for exact matching files.
I will take a picture later today of it and show you they are the same. at least the data shown is.
Title: RE:[Plugin] Duplicate finder
Post by: Mirko on September 17, 2002, 09:45:18 am
@KingSparta:

I believe you. But I'm interested in a screenshot. Maybe you should send it to my private adress (mwalter@2via.de).

Thanks a lot,
Mirko
Title: RE:[Plugin] Duplicate finder
Post by: KingSparta on September 17, 2002, 10:12:25 am
Will Do, By The Way How Do You come Up With The Percent?

If It Is By File Name That Maybe Why The Percent Is Off.

I Will Scan Now And Send It To You, I Also Had A Problem With The Song Play Time Was Wrong On Some of The Files, I Will Try To Recreate That Also. It basicly was the seconds were correct but the mins were 00:14 when it should have been something like 03:14

I Just did a run this time I got 75% but the file size, bytes, and Kbps match. Picture on the way.

that was for version 1.0.251 i see you have a new version out.

i will see if the problem is still there
Title: RE:[Plugin] Duplicate finder
Post by: KingSparta on September 17, 2002, 11:05:21 am
Just checked the new version problem is still there

Now I have 75 files that = 100%

Besides deleting them manualy how do I get them Automaticly into the Dupster?

It seems the problem happens when you have "files with less duration" checked.
Title: RE:[Plugin] Duplicate finder
Post by: Mirko on September 17, 2002, 11:22:22 am
@KingSparta:

Did you try using the right mouse button? It opens a contextmenu where you can choose "autodeletion..." or something similar. There you can select "delete same files". This should do the trick I think.

HTH,
Mirko

P.S. The mail is not relevant anymore, is it?
Title: RE:[Plugin] Duplicate finder
Post by: KingSparta on September 17, 2002, 11:32:54 am
>> It opens a contextmenu where you can choose "autodeletion..." or
>> something similar. There you can select "delete same files". This
>> should do the trick I think.
yep, no go

they say 75% but they all match all areas Bytes, Kbps, and File Size

playing with it some more.

Try this, set the Duration from the Default "2" Seconds To "0" thats where it seems to go bad, and then It gives the wrong Percent
Title: RE:[Plugin] Duplicate finder
Post by: KingSparta on September 17, 2002, 11:41:09 am
After I set it back to 2 seconds I get 100% for all of them but one.

that one seems to have a slight dif in the song name

Love story & Love Story (Where Do I begin)

so i do not understand why that Precent should change if i change the seconds in duration if the Bytes, Kbps and duration is the same it should be 100%.

On a happy note:

By the way, this version is screaming with speed!
Title: RE:[Plugin] Duplicate finder
Post by: Mirko on September 17, 2002, 12:14:56 pm
KingSparta:

As far as I know you develop yourself in vb. So here's the coding:

   If Trim(A1) <> "" And Trim(A2) <> "" Then
       bA = (InStr(1, A1, A2) > 0 Or InStr(1, A2, A1) > 0)
   Else
       bA = False
   End If
   If Trim(T1) <> "" And Trim(T2) <> "" Then
       bT = (InStr(1, T1, T2) > 0 Or InStr(1, T2, T1) > 0)
   Else
       bT = False
   End If
   
   qFS = IIf(FS1 > FS2, CDbl(FS1) / CDbl(FS2), CDbl(FS2) / CDbl(FS1))
   qD = IIf(D1 > D2, D1 - D2, D2 - D1)
   qK = IIf(K1 > K2, K1 - K2, K2 - K1)

   Q = IIf(bA, 1, 0) |PLS| IIf(bT, 1, 0) |PLS| IIf(qFS < cMatchDiff, 1.5, 0) |PLS| IIf(qD < Val(txtDurationOff.Text), 1.5, 0) |PLS| IIf(qK < Val(txtKBitOff.Text), 1, 0)

As you might see, I calculate the quote by Artist, Name, Filesize, Duration and Bitrate. The "cMatchDiff" for Filesize is a constant about 1.015. The rest are the values entered into the textboxes in the setup-tab. If you have differences in filesize e.g., then you would not get a 100% quote.

Any suggestions?

Edit: I updated the plugin with a new autodeletion-option "delete same files matching in size,duration,kbits". Maybe this will do what you like.
Title: RE:[Plugin] Duplicate finder
Post by: KingSparta on September 17, 2002, 12:46:39 pm
>> Any suggestions?
Looks Complex

What i do know is if i change the duration in setup from "2" to "0" that should have nothing to do with the Percent between the two files your trying to match.

The setting in setup should be a tolerance

if it = 100%duration to Duration or |PLS| or - set seconds

So if the file was 3:30 to 3:32 or 3:30 to 3:28 the file should match. and it should have nothing to do with the percent or the total Precent of the file.
Title: RE:[Plugin] Duplicate finder
Post by: Sam. on September 17, 2002, 01:38:15 pm
Mirko,  This is awesome!!
826 dups out of 15,179.  In about 6 minutes.
To think, I was going to try to find them all manually!

The user interface is great too.  You should work on MJ9. :)


a couple things...
In the summary, it says 826 dups.  At top, it says Duplicates found: 825

The pop-up note when the mouse is over the bottom box says, "Doublicates selected for deletion".  I know this community spans the globe, so I won't assume that this is not how to spell it elsewhere... :)

Out of a few hundred 100% matches, there were 4 pairs that had non-matching album names.  In each of the 4 pairs, one of the files had a blank album name.

Clicking on "Delete!" brings up a dialogue box.  The title says "Move duplicates to playlist."  The text says "Remove the files from the duplicates-playlist?"

When I play a song in the dup list and I click anywhere in the dup info window, the window goes blank.

The instruction that pops up after "Delete!" says that you can delete songs from the playlist by pressing [Shift]|PLS|[Delete].  I remember being able to delete from a playlist by pressing [Shift] and right clicking on "Remove".  Well, either way, I can't seem to delete the song.  It'll just be removed from the playlist.


That's all for now.  And again, I love this.

Sam
Title: RE:[Plugin] Duplicate finder
Post by: Mirko on September 17, 2002, 02:23:17 pm
@Sam:

Out of a few hundred 100% matches, there were 4 pairs that had non-matching album names. In each of the 4 pairs, one of the files had a blank album name.

Bug-Alarm! Albums were not included in the match-quote-calculation. I have fixed that.


When I play a song in the dup list and I click anywhere in the dup info window, the window goes blank.

I have the same problem here. And have no idea how to fix that. I thought it might have to do with skinning... I already did some pictureboxes around the questionable elements, but nothing (I read this reading ChartFinder-threads).

@KingSparta:

I had a little error in the match-quote-calculation. The match was "qD < val(txtDuration.Text)" but should have been "qD <= val(txtDuration.Text)". I corrected this. Could you please check it again? I have no files matching those criterias or maybe I'm not smart enough to figure them out :-)

Thanks for the feedback,
Mirko

P.S. Updated :-)
Title: RE:[Plugin] Duplicate finder
Post by: KingSparta on September 17, 2002, 03:11:01 pm
Mirko

Should you not be Sleeping? or did you just get up?

I will need to download some test files since the last version I was able to Clear things up around this maze of files I have.


'I Can See Clearly Now' Charted At 01 In 1972

Listening to: 'I Can See Clearly Now' from 'The Best of Johnny Nash' by 'Johnny Nash' on Media Jukebox
Title: RE:[Plugin] Duplicate finder
Post by: Mirko on September 17, 2002, 08:39:20 pm
@KingSparta:

I should have been sleeping... got up about 04:45 and went to bed about 00:45 (that are 24h-time... I think 04:45 am to 00:45 am may be the right translation). And got up 04:45 today :-/ Yes, maybe I'm a little tired today ;-)

I did another little update before I went to sleep. The "label1"-thing still showed "label1" on startup. Today I will try to optimize some more, I think there is some potential in the way I modify the listviews (I show them to the user during processing, maybe disabling update will speed up a little).

How long does the plugin take on your collection? If I remember correctly, you have about 40.000 songs, so I would be interesting, how fast the new algorithm is. I think I can speed it up a little, if I do a second index (this will complicate things dramatically, but will help searching collections that big... will think about it).

Bye,
Mirko

@Sam:
I repaired all things except the "turning white"-thing. I will care about that later.
Title: RE:[Plugin] Duplicate finder
Post by: nila on September 18, 2002, 12:20:52 am
Mirko - could I possibly have another Nila Special.
For autodelete options I'd like: Album name = Random Songs :)
Basically for it to just delete whatever is set under the other Nila Special Option :)

Thanks :)
Title: RE:[Plugin] Duplicate finder
Post by: Mirko on September 18, 2002, 12:44:30 am
@Nila:
Did you recently look into the new dialog? ;-)
The option is (was, will stay) there.

Edit: I will update the plugin this evening. I think there is a bug in the memory-index-function, that may lead to not finding duplicates with articles ("The", "A") in the names. The routine for stripping those things is ready for action but not included yet.
(e.g. if you filed "Beatles" and "The Beatles" the plugin will not evaluate it as matching).
Title: RE:[Plugin] Duplicate finder
Post by: graham131 on September 18, 2002, 12:52:02 am
Mirko,

Hi I am very interested in this plugin - however whenever i download it from the web via MJ it downloaded but fails to show up in my Plugins list.

Any ideas?

Graham
Title: RE:[Plugin] Duplicate finder
Post by: Mirko on September 18, 2002, 12:53:39 am
@graham:

Did you download from the interface-plugins page? Did you start the plugin (look in the plugin-dialog and press the "start" button -> this occurs after a restart of MJ automatically)?
Did you click "ok" the prompt which asks you do integrate something into the registry (you have to)? If this prompt doesn't show up, try downloading it from
here.
Title: RE:[Plugin] Duplicate finder
Post by: graham131 on September 18, 2002, 01:26:30 am
downloaded your resistry link and ran it.  This now recognises the Plugin but whenever MJ starts or i click on the plugin i get the following error message.

"Interface Plug-in'DupFinder' must implement |Unknown interface"

Thanks

Graham
Title: RE:[Plugin] Duplicate finder
Post by: Mirko on September 18, 2002, 01:48:24 am
Graham:

Do you have the VB6-runtime files installed? These are necessary (please take a look at the interface plugin chart finder description, KingSparta provided a link to these).
If this all doesn't help, please be patient. I'm currently at work and can't look into it in detail. But I will.
Title: RE:[Plugin] Duplicate finder
Post by: graham131 on September 18, 2002, 02:36:56 am
Thanks I will try that, however i am already running the king's chartfinder plug in sucessfully so would assume that I have these necessary files already

Cheers

Graham

PS I appreciate the help and prompt responses
Title: RE:[Plugin] Duplicate finder
Post by: Mirko on September 18, 2002, 02:54:29 am
Graham:

I don't know how experienced you are with computers. But in case you are: There are some files in the ZIP-archive for my plugin that end on .OCX. Maybe some of them are not registered correctly. You could test it by running "regsvr32 [name].OCX" on them (doubleclick and choose the regsvr32.exe from the system32-directory below \windows to run with).
If that doesn't help either, I think the last build is not working. Did anybody else got this error during the last few builds?

Thanks a lot for helping finding errors.
Mirko
Title: RE:[Plugin] Duplicate finder
Post by: nila on September 18, 2002, 03:32:51 am
Mirko,
the option is there, but what I meant was an auto delete option too so that for all the songs that it found using my special rule, it would automatically select the one that had the album name set as 'Random Songs' as the one to delete.
Title: RE:[Plugin] Duplicate finder
Post by: Mirko on September 18, 2002, 03:39:30 am
Nila,

aaaaaaahhhhh... *bang* (did you hear the cent falling?)

I will do this. Later ;-) (btw. why don't you make a smartlist for that?)
Title: RE:[Plugin] Duplicate finder
Post by: KingSparta on September 18, 2002, 03:43:59 am
graham131

Make sure you Have the latest version of MJ8, the update was posted yesterday.

if you have not installed visual basic runtime files that may help. not all programs install all the VB runtime files or dlls that maybe needed to run a VB project (it can't hurt).

reboot, do not load MJ

go to the plug-in's page and download the plugin and install.

load MJ see what happens.
Title: RE:[Plugin] Duplicate finder
Post by: KingSparta on September 18, 2002, 03:52:12 am
Mirko

Some Ideas:

In the last version if you use the delete all 100% matches it works, but on the scan you do not have an option for this to Automaticly delete the 100% matchs.

-----------------

Howabout an Option

[X] Clear Delete List On New Scan

Also A [Clear] Button


-------------------

When You Press Play 1, Change That Button To "Stop", Leave Play 2

this way a user does not need to press "Stop" and then "Play 2"

-------------------

Press Delete

Asks Do You want to remove files from match list, if you press no it does it anyway.

-------------------
If you press "Save" the Header for the MsgBox Is Cut Off
----------------------
If you press "Load" the Header for the MsgBox Is Cut Off
----------------------

Sometimes when you select a song in the match box the box seems to clear.

-------------------

If you dbl click on a file in the delete list play it.

-------------------

if you select the file in delete also highlight the files in the match list.

---------------------

when you move from one file to the next (maybe using th up and down arrows) autostop play and auto start play.

---------------------

auto play time, in setup combo box user selects will play song 1 for lets say 15seconds, then stops and plays the second song for 15 seconds)
Title: RE:[Plugin] Duplicate finder
Post by: nila on September 18, 2002, 04:55:46 am
How can I use a smartlist for this?
I need your plugin to find the duplicates, if I could then export all these files in full to a playlist straight away then I could use a smart list but there's no option to do that that I'm aware of :(

Might be a nice option to just add all the files straight to a playlist if we wanted.
Title: RE:[Plugin] Duplicate finder
Post by: Mirko on September 18, 2002, 05:34:42 am
@Nila:

I thought about a filter like "album=[random songs]" in a smart-playlist. So you can always see the songs, that are currently assigned to this album.
Isn't this what you would like?

Sure I can add that option. I was just thinking, that I would solve this problem another way.
Title: RE:[Plugin] Duplicate finder
Post by: graham131 on September 18, 2002, 06:10:40 am
Mirko.

Thankyou thankyou thankyou

By using the Regsvr32 and registering individually it is now working.

I will now get on with the importnat bits of playing with and testing it.

Cheers

Graham
Title: RE:[Plugin] Duplicate finder
Post by: graham131 on September 18, 2002, 06:29:12 am
WOW!!!!!!

24,500 Tracks scanned in 13 mins - 813 Duplicates found however I do need to now refine my searching a bit as I do want to keep dupes on certain albums etc ie Greatest Hits.

This is what I have been waiting for.

Along with Kingsparta (for Chartfinder) I would like to vote Mirko MJ'er of the week!!!!!!

Regards

Graham
Title: RE:[Plugin] Duplicate finder
Post by: KingSparta on September 18, 2002, 06:33:50 am
graham131

Can you state step by step what you did and what you entered.

there are other people having this problem, and maybe with your help we can sort out what and how users can register both chart finder, and Mirko's program "Duplicate Finder"
Title: RE:[Plugin] Duplicate finder
Post by: nila on September 18, 2002, 06:34:28 am
That would show me all the songs that were in the 'Random Songs' album but it wouldn't show me all the duplicates that had their album set to Random Songs :).

Finding the duplicates that exist is the key here :)
Title: RE:[Plugin] Duplicate finder
Post by: KingSparta on September 18, 2002, 07:08:59 am
Nila

If you make a play list with random songs.

right click on "Go" this will allow you to select your new Random Song Play List, then you can run Dup Finder on that list.
Title: RE:[Plugin] Duplicate finder
Post by: Mirko on September 18, 2002, 07:33:16 am
It seems like KingSparta does a fine job in supporting the plugin ;-)

I think that what graham did will help many others. Maybe we should make a MJP-file for "Visual Basic runtime - plugins". Then we could contribute this to the interface plugin section on JRiver website. What do you think?

Thanks a lot.

Now I quit working for my customers and try finding my way home. If the autobahn is not full of cars today like it was yesterday, a "little" accident, I can begin adding the requested features in about an hour :-))

Mirko
Title: RE:[Plugin] Duplicate finder
Post by: KingSparta on September 18, 2002, 07:44:20 am
>> I think that what graham did will help many others.
yep

> Then we could contribute this to the interface plugin section
> on JRiver website.
Reading My Mind Again?


I finaly ran into a German Beer I hate last night.

Weihenstephaner - Hefeweissbier Dunkel (Sucks)

The other 4 or 5 Beers they make are good
Title: RE:[Plugin] Duplicate finder
Post by: nila on September 18, 2002, 08:51:56 am
ok,
Tried the smart list, it doesn't work.
That just finds duplicates within my random songs bunch.

Basically the Nila special option was what I wanted.
The only extra thing I wanted was to be able to automatically delete the matching song that had the album name specified in the 'nila special'

Still not following me? :(
Title: RE:[Plugin] Duplicate finder
Post by: Mirko on September 18, 2002, 10:41:28 am
@KingSparta:

I can _really_ understand why you didn't like that beer. Really. I'm coming from the Rheinland, which means we typically drink "Königspilsner" (Pils), "Kölsch" (a beer from cologne) or "Alt". My favorite is Altbier, exactly "Frankenheim", a small local brewery (which exports to the world I think).

Your ideas were very good. I integrated all but one of them (I have to implement the buttons for clearing the listviews).

@Nila:

I did it ;-) The update is on it's way.
Title: RE:[Plugin] Duplicate finder
Post by: KingSparta on September 18, 2002, 11:35:02 am
Bug 1:

When you copied the new Options in setup, you did not change the Tool Tips for the last 3 options.

Bug 1:

On The version page some of the text is cut off.

There must be a Clearer way to show a user what file is playing.

Maybe In The Difference Info Box?

-------------

Just a Note about the screen slearing on the match list. with MJ skins Disabled it works better. i have not seen this problem on the delete list. maybe it is someting your doing or not doing to the match list that is creating this problem with the list clearing (Resizing? something else?) when a user clicks on the box.


Now Drinking "Hoegaarden" The Orginal White

A Naturally Cloudy Non-Filtered, Belgium Beer
Title: RE:[Plugin] Duplicate finder
Post by: Mirko on September 18, 2002, 12:29:20 pm

Bug 1:
When you copied the new Options in setup, you did not change the Tool Tips for the last 3 options.
Bug 2:
On The version page some of the text is cut off.



Fixed.



There must be a Clearer way to show a user what file is playing.
Maybe In The Difference Info Box?



I have found a little source for adding icons to subitems in the listview. So you now can see an ear listening to the song ;-)

Mirko, now going to bed, who only knows belgium schokolade.
Title: RE:[Plugin] Duplicate finder
Post by: Sam. on September 18, 2002, 01:02:06 pm
Mirko,
I know it must be difficult receiving suggestions from users and trying to make everyone happy.  You're doing great.  Just remember that you can't please everyone.  :)  And feel free to say 'No' sometimes.

I've been trying to think about how I would choose which songs to delete after DupFinder identifies the matching songs.  I have a few suggestions.  (Some of them are big...)

I don't think the percent calculation is very useful to me.  If it's an exact match, then great, I'll delete either song.  But if the match is less than 100% (whether it's 99% or 33%), I'll want to know what exactly the difference is before I choose to delete a song.  The number itself doesn't help me make a decision.

Similarly, I can't choose most of the Autodelete options because there will always be exceptions.  For example, if the songs are from two different albums, I would not want DupFinder to automatically delete the shorter song.  I suggest that the Autodeletion options be limited to specific instances.  For example, delete the shorter song only if the song length is the only difference (meaning that the album names are the same).
Perhaps like this:
[X]  Exact match - Delete either one.
[X]  Exact match except bitrate (and file size) - Delete song with lower bitrate.
[X]  Exact match except length (and file size)- Delete shorter song.
[X]  Exact match except file size - Delete shorter song.
[X]  Exact match except one album name is blank - Delete the song with the blank.


Can you have DupFinder identify which song to delete based on user preferences and make that song #2 in the top window (or #1, either way)?  The user just has to "approve" the recommendations made by DupFinder based on the rules set by the user.  These rules could be the Autodelete preferences above.
Currently, Autodelete copies the songs to be deleted to the bottom window.  It's difficult for the user to do anything with that.  It would be easier if the song data were next to each other in the main window.  


And can you highlight what the difference is between the two songs?  For example, if the artists and titles and lengths are identical, maybe we don't need to see them twice?  We just need to see that one is recorded at 160kbs and the other at 128kbs - and that DupFinder suggests deleting the 128kbs file.


Also, how broad is the original search?  I think I have many duplicates in my library where the artist is the same, but the titles are different:
One title might include "Live" or "a capella" or "remix."  
One title may be truncated or appended (I Say A Little Prayer" vs. "I Say A Little Prayer For You.")
Many titles include the artist name, album, and track #.
Could DupFinder identify these songs?

Thanks.  (And feel free...)


Just trying to help,
Sam
Title: RE:[Plugin] Duplicate finder
Post by: KingSparta on September 18, 2002, 01:32:28 pm
When Playing Icon Shows On Artist1 And Artist2

Should That Not Be On Artist1 And Name1?

Maybe A Standard ">" Play Icon When It Is Playing

Looking Good...

Whats Next?

Save Config?

First Line In My Config Is App.Version Just to make sure Nothing Has Changed And If Version Does Not Match Reset To Default.
Title: RE:[Plugin] Duplicate finder
Post by: Mirko on September 19, 2002, 06:22:39 am
@Sam:

Some very nice ideas. I have to think them over.


And can you highlight what the difference is between the two songs? For example, if the artists and titles and lengths are identical, maybe we don't need to see them twice?


I did a little something here. You get an icon in the columns, that match. This is not my favourite solution, I would like to have bold or italic fonts here, but the only one known to me yet.

@KingSparta:

The plugins saves options now. Automatically, so there is not button or something like that.
The bugs are fixed.

I'm on a little vacation the next three or four days, so there will be no update until then.

Mirko
Title: RE:[Plugin] Duplicate finder
Post by: KingSparta on September 19, 2002, 06:41:54 am
>> I'm on a little vacation the next three or four days
>> so there will be no update until then.
don't you think you have that backwards?

I am on vacation, and my wife has me painting, Planting trees etc...
Title: RE:[Plugin] Duplicate finder
Post by: Mirko on September 19, 2002, 07:24:39 am

don't you think you have that backwards?


I don't understand that (as a non english-speaker I may have missed the point)...


I am on vacation, and my wife has me painting, Planting trees etc...


I know that kind of vacation :-/
Title: RE:[Plugin] Duplicate finder
Post by: KingSparta on September 19, 2002, 08:36:40 am
Mirko

>> don't you think you have that backwards?

> I don't understand that (as a non english-speaker
> I may have missed the point)...
Joke

Normaly My Vacations Are Filled With Work My ife Has On A List, And She Is Checking Them Off As I do Them


Now Drinking: Heineken, Special Dark Beer

"Beer" The Gift Of Life

Recently It Was Found That The First Thing Cave Men Created Was Beer (Prob To Cope With The Cave Wife).
Title: RE:[Plugin] Duplicate finder
Post by: Mirko on September 19, 2002, 08:54:21 am
I added what Sam suggested. Not in perfection (you have to do it one after another, not multiple at once -> I would like that *g*), but they are there. You were right, it helps.

But about the deletion list... I like it. So I can easily check again, if this was the right decision. Using the feature KingSparta suggested (automatic selection of the corresponding item in the duplicates list), it is very helpfull in comparing. So this will stay (or, like you, I say "no" ;-) )

Ok. Now I have to help packing the things for the weekend. "Helping" is a little bit too much. I have to listen to things like "I don't have enough cloths, you see that, don't you?" - "Err... (thinking: my own things are stuffed into 20cm, she has about 2meters) Yeah babe, you're right *sigh*".

Mirko
Title: RE:[Plugin] Duplicate finder
Post by: KingSparta on September 19, 2002, 09:11:26 am
>> my own things are stuffed into 20cm, she has about 2meters)
>> Yeah babe, you're right *sigh*".
I feel your Pain
Title: Re: [Plugin] Duplicate finder
Post by: Mirko on September 19, 2002, 06:01:05 pm
I just did a new update...
Title: Re: [Plugin] Duplicate finder
Post by: kandarohi on September 21, 2002, 12:50:21 am
i've just waded through this thread, and although i did not understand most of what i read, i like the idea of a duplicate finder/eraser for mj. can someone please post a weblink to where i can download the latest version of this plugin, or tell me where to find it? if it is a long address, i would need a direct link, as my 1 finger typing is so innaccurette. thanx.
Title: Re: [Plugin] Duplicate finder
Post by: wickesy on September 21, 2002, 03:22:17 am
You can get it from www.2via.de   :)

All the best

David Wickes
Title: Re: [Plugin] Duplicate finder
Post by: KingSparta on September 21, 2002, 03:57:24 am
It Is Also In The MJ Plug-Ins Section

http://www.musicex.com/cgi-bin/downloads/plugins.pl?type=10&start=0&end=10&page=1
Title: Re: [Plugin] Duplicate finder
Post by: zak326 on September 21, 2002, 02:17:19 pm
This plugin really is a great idea and all the tweaks you guys are suggesting sound like a good end result but at this point they might as well be written in Swahili.
\Are there any clear instructions on how to use this thing.  I've been ex[perimenting wqith it , but have had mixed results.

I know that normally men dpn't read directions or lookl at maps until they havescrewed something up or are lost. I am now a bit of both. Help?
Title: Re: [Plugin] Duplicate finder
Post by: Charlemagne 8 on September 21, 2002, 02:31:05 pm
Mirko,
I have the same problem with this one that I had with King's. No registry writing. Is this restricted to Non-Win98 OS versions also?
CVIII
Title: Re: [Plugin] Duplicate finder
Post by: KingSparta on September 21, 2002, 02:47:43 pm
if you use a plugin it basicly uses the same Registry.reg file with some small changes (File Name, Version Etc...) as the J river SDK.

I Asked The Question On Maybe how to fix this or if there was a work around for users with An Older OS.

But i have never been given an answer or the time of day. It Seems Plug-Ins are not a priority right now and Version 9 is, And i understand to a point.


Title: Re: [Plugin] Duplicate finder
Post by: Charlemagne 8 on September 21, 2002, 03:19:59 pm
And so we wait. It will be resolved sooner or later. I am actually toying with the idea of getting a newer computer. Mine is two years old. That's old for a computer. I don't really have a problem with getting a new computer, just paying for one is the sticking point.
CVIII
Title: Re: [Plugin] Duplicate finder
Post by: Mirko on September 22, 2002, 01:52:20 am
@zak326
Quote
Are there any clear instructions on how to use this thing.  I've been ex[perimenting wqith it , but have had mixed results.


It is quite simple to use, I thought: I added some instructions to the "About/Help"-Tab.

But these are the simple steps:


If there are any other question, please ask again.

Mirko, vacation to short, life is hard.
Title: Re: [Plugin] Duplicate finder
Post by: Sam on September 23, 2002, 07:11:29 pm
Does the plug-in decide which song to be #1 and #2, or is it random?  It could be helpful if one was always the smaller, shorter, or lower bit rate one.

And what is that squiggly thing on the left?   :)
Title: Re: [Plugin] Duplicate finder
Post by: Mirko on September 23, 2002, 08:22:59 pm
Sam,

I don't know if this the order #1 or #2 is random. MJ gives me the whole database and I go through it (one by one, more or less). So the one songs that comes first, gets first.

I have to think about how to order the songs. You are the second IIRC that requested such a feature. But I'm not sure, how to do this. If the bitrate and duration and filelength are different, there is no easy choice which one should be significant for ordering. Ok, I might do this as an option, but if I do this, some time in the future the setup-tab gets bigger than the actual function-tab ;-)

What do you meen by "And what is that squiggly thing on the left?"? Do you mean the percentages? These are matchquotes, e.g. marks on how matching to songs are (if mq is 100% they are the same, if it is 80% they are nearly the same).
Title: Re: [Plugin] Duplicate finder
Post by: Sam on September 24, 2002, 06:28:15 am
No, not the percentages.   LOL

Here it is...  
(http://samuelkim.com/public/dupfinder/dupfinder%20screencap.gif)


That's okay; I don't need a "Sam-special" option in the tab. ;D
Title: Re: [Plugin] Duplicate finder
Post by: LCtheDJ on September 24, 2002, 08:50:26 pm
Mirko,

I just installed your Duplicate Finder plugin and have not yet become familiar with all that it can do, but I am wondering if it can help with this:

I'm re-ripping my CDs and encoding them in Ogg Vorbis to replace the mp3 copies of the same recordings.  Can the Duplicate Finder find files with the same filename except for the file extension, and then delete one of them?

In other words, I want it to find and delete "Filename142.mp3" when there is also a "Filename142.ogg".

Can your plugin do this?  If not, could this function be added to it?
Title: Re: [Plugin] Duplicate finder
Post by: Mirko on September 24, 2002, 09:00:46 pm
It doesn't do this right now.

But, sure, I can add this "autodelete-option" (some sort of preselecting of to deleted songs).

Will have to wait for a min. of 10hours (I'm currently at work).

Just to clearify: The filenames are _exactly_ the same, except extension (upper/lower case is ignored)? There are no little things like track# added and such? Spaces are ignored at this moment, so adding spaces doesn't matter.

Title: Re: [Plugin] Duplicate finder
Post by: Mirko on September 24, 2002, 09:04:07 pm
Quote
That's okay; I don't need a "Sam-special" option in the tab.


The problem with this little white squares is, that I don't put them in on purpose. It's a side effect of the mechanism I use to add icons to listview-items. In fact, there _are_ little icons in this white squares. But the transparency of the used colors is not correctly applied (and I'm in fact no sort of artist (and lazy), so I (maybe you) have to look for a 16x16 ICO-file without any contents but transparency bits).

Maybe someone could provide such an icon. If anybody has one, please send it to mwalter@2via.de, put please zips them (they are sooooo big :D ).
Title: Re: [Plugin] Duplicate finder
Post by: JimH on September 25, 2002, 05:35:23 pm
Quote
But i have never been given an answer or the time of day. It Seems Plug-Ins are not a priority right now and Version 9 is, And i understand to a point.


King, I'm sorry, but I missed this one.  It is 8:35PM.