INTERACT FORUM

Windows => Television => Topic started by: CountryBumkin on September 02, 2015, 09:46:05 am

Title: using XMLTVDB to enhance xmltv.xml data
Post by: CountryBumkin on September 02, 2015, 09:46:05 am
I'm trying to get the program "XMLTVDB" (version 1.0.1) to work to add Season Numbers to the xmltv.xml data coming from mc2xml (Micosoft Rovi or Schedules Direct).

The program is supposed to monitor the (EPG data) file "xmltv.xml" for updates and when an update is detected it goes out and looks for additional EPG data (such as Season Number and Episode Number) and then creates a new file named "xmltv.enhanced.xml" which would then be imported into JRiver MC.

The XMLTVDB program has not been in development since 2012. It looks like it uses the source "ForTheRecord" to collect data from. However the FTR website is no longer functional. The code uses "http://FTR:49943". This is deadend from what I can tell.

There is another XMLTVDB program (version 1.1.1) that uses "ForTheLibrary", however this version of the program does not have folder monitoring so it doesn't appear to be of any use.

When I run the XMLTVDB program, it appears to detect the changing xmltv.xml file, but then it does not create xmltv.enhanced.xml file. I think the data source of FTR is no good. If anyone has this working - where is the enhanced data coming from?

Note the "FTR_WEBSERVICE_URL=http://FTR:4993".  Is this the same as your (working) program?


Thanks.
Title: Re: Is anyone running XMLTVDB to enhance xmltv.xml data?
Post by: CountryBumkin on September 02, 2015, 05:06:34 pm
This is from RammingSpeed:
 (I hope you don't mind my moving/copying your post)
Quote
I got XMLTVDB working and here are a list of things to look out for.
1.) Use Java 7 (6 won't work, didn't try 8 )
2.) Make sure you are using the latest paid version of XMLTVDB mc2xml.  There was a previous release that worked with Rovi, but didn't handle ampersands correctly in the XML.
3.) review the "XMLTVDB.properties" file and change settings correctly.  There is documentation in the file
4.) If possible set up EXCLUDED_CATEGORIES as it will reduce the amount of lookups and time to complete.
5.) Consider setting "HTTP_REQUEST_RETRIES=" to a smaller number
6.) If you need to iterate testing, change the timestamp on your XML file by re-saving it.  XMLTVDB will pick that up as a new file.

Note that it takes a long time to complete, so be patient once it starts.  You'll see lots of errors in the window.  As long as it keeps running, everything is working.

I'm using java 8. I'm not using the "paid" version of mc2xml because I'm using the "paid" Schedules Direct as my data provider (not Rovi). Perhaps this is an issue.

So, how do you know if it's running?
I have no "cmd window" to view the program while its running.

I just know it's not creating the "xmltv.enhanced.xml" file. But I'm not sure why (although I think the link to use the "FTR" website may be the issue).
Title: Re: Is anyone running XMLTVDB to enhance xmltv.xml data?
Post by: RoderickGI on September 03, 2015, 12:09:06 am
09/01/2015 12:04:12 PM DEBUG   ONLY_LOOKUP_SCHEDULED_SHOWS = true

For a start, set the option for this line to false. You aren't (can't any more) using integration with ForTheRecord, so XMLTVDB can't look up what you have scheduled.
Code: [Select]
#Set to true to only lookup shows that you have scheduled to record in ForTheRecord. (Recommended true)
ONLY_LOOKUP_SCHEDULED_SHOWS=false

I would also recommend that you comment out the line referring to ForTheRecord.
Code: [Select]
#The URL for 4TR's web services. Default port is 49943
#For basic authentication, use http://username:password@FTRSERVER:49943
#FTR_WEBSERVICE_URL=http://FTR:49943

ForTheRecord and ForTheLibrary also referred to in the description of XMLTVDB were basically the same thing I believe, or components of a solution set. Now defunct, they have morphed into ArgusTV I think. There are lots of references to ForTheRecord in ArgusTV forum posts. I can't really tell if there was just a renaming, or ForTheRecord was just used by ArgusTV at some stage.

Note that I haven't tested this myself, but it just makes sense. You want to look up all programs that are in the XMLTV file on TheTVDB site.
Title: Re: Is anyone running XMLTVDB to enhance xmltv.xml data?
Post by: RoderickGI on September 03, 2015, 01:09:42 am
I did a test run using one of my old IceTV XMLTV source files, just to see if it would work. With the above option set to False, XMLTVDB ran fine.

My source XMLTV files was 3,139KB, and the output file was 3,497KB, so quite a bit of data was added. Checking through the output, the first thing I noted was that it formatted the output file better than the original IceTV file, so that it was now readable using Notepad. The source didn't have line breaks and I had to view it in Explorer. Yuck. Notepad is much easier, particularly when searching for fields.

It seemed to do a pretty good job of finding the Season and Episode numbers for all well known America TV shows, where the source file had both the <title> and <sub-title> fields included. It wrote the results to the output file in xmltv_ns format. For local Australian shows it didn't work very well, and for any that didn't have the <sub-title> field it didn't work at all, that I could see. I didn't check if it did any fuzzy lookups where names weren't a perfect match. I believe some of the tools written for ArgusTV and other products can do fuzzy lookups. At least some of the descriptions imply that, such as for "UltraXMLTV Enhancer", which also seems to be a dead product.

I used version 1.0.1 of XMLTVDB. It used 1% to 3% of CPU, mostly just 1%. It ran 18 threads, downloaded at around 25KB/s, and cached over 1100 files during the process. I used Windows Task Manager and Resource Monitor to watch all that. Theoretically you should be able to have the Java Console open when you run any Java program, and you can watch what it is doing in there as well. I didn't this time.

The log included the lines below.

09/03/2015 03:29:49 PM NOTICE  Total: 1190 successful lookups, 893 failed lookups.
09/03/2015 03:29:49 PM NOTICE  Of 4112 total programs, 623 unique series were looked up.
09/03/2015 03:29:49 PM NOTICE  Performed 1323 total TheTVDB.com queries (11.87 MB of data), and 3185 locally cached TVDB.com queries

All that took about 10 minutes to run.

So I guess XMLTVDB would work as a method of adding Season and Episode numbers to an XMLTV file that lacked them. I couldn't find any independent tool that was still being supported, which would have been better. But it works, for now. As long as TheTVDB doesn't change its API, which apparently it hasn't for quite a while.
 
Title: Re: Is anyone running XMLTVDB to enhance xmltv.xml data?
Post by: CountryBumkin on September 03, 2015, 05:14:09 am
Thanks again for help.

I commented out "FTR" (I thought that FTR was the source of the enhanced data.) - So the program instruction to look for enhanced data at TVDB must be in the java code. I don't see any reference to TVDB.com in the xmltvdb.properties/settings. I know nothing about java code.

I changed the "true" to "false".

I'm still ending with the "DEBUG   LISTENER: Skipping Event ENTRY_MODIFY. File is: xmltv.xml".

I'll try again with the Java Console open.

When you run the program, are you getting a new file called "xmtlv.enhanced.xml"?

 I think I am the only one who cares about getting Season Numbers. Maybe I just need to convert my TV series database from Season Number to Season Year and move on.
Title: Re: Is anyone running XMLTVDB to enhance xmltv.xml data?
Post by: RoderickGI on September 03, 2015, 05:53:26 am
FTR was a TV recording application, long gone now. So XMLTVDB was trying to check it to see what was on the recording schedule, and then only check TheTVDB for those programs, as a way of reducing load on TheTVDB.

Yep, I get a nicely enhanced xmtlv.enhanced.xml file.

The last part of my log is;

09/03/2015 03:29:49 PM NOTICE  Performed 1323 total TheTVDB.com queries (11.87 MB of data), and 3185 locally cached TVDB.com queries
09/03/2015 03:29:49 PM NOTICE  --------------------------------------------DONE PROCESSING XMLTV FILE--------------------------------------------
09/03/2015 03:29:49 PM DEBUG   Not sending e-mail notification because it is disabled in config file.
09/03/2015 03:29:49 PM DEBUG   LISTENER: Skipping Event ENTRY_MODIFY. File is: icetv.xml
09/03/2015 03:29:51 PM NOTICE  LISTENER: Processing xml file: xmltv.enhanced.xml
09/03/2015 03:29:56 PM INFO    Logger has been initialized after a reset
09/03/2015 03:29:56 PM NOTICE  Processing: F:\Junk\MC\XMLTV\xmltv.enhanced.xml

As you can see in the last line, it tries to process the enhanced file, because I put the enhanced file in the same location as the input file. If I hadn't done that I think the log would have ended at the bolded line, which is XMLTVDB looking at the input file a second time, seeing it is the same as last time, and that is probably when it renames it to "icetv.xml.old", which is the file left in my input directory. So my log would end with the same line yours did.

If you aren't getting the output file, check your directory specifications carefully. Make very sure that your "XMLTVDB.properties" file doesn't have a .txt extension on it, hidden or visible. Also, make sure that it is saved in UTF-8 format, and not ANSI, using the File/Save As menu item in Notepad.

I think other people care, a lot, but perhaps not as many MC users as Kodi, ArgusTV, NextPVR, etc. users. The forums are full of people trying to get and improve EPG data. If you change to using Year instead of Season number, somewhere down the track it will bite you, and all you records will be wrong in MC. I would persist with XMLTVDB, or use EPG Collector to do the same thing, if XMLTVDB won't work for you.

PS: I'm attaching a text version of my "XMLTVDB.properties" for you to compare. Note the full directory specifications, which I have now changed to separate Input and Output locations.
Title: Re: Is anyone running XMLTVDB to enhance xmltv.xml data?
Post by: CountryBumkin on September 03, 2015, 10:39:30 am
check your directory specifications carefully. Make very sure that your "XMLTVDB.properties" file doesn't have a .txt extension on it, hidden or visible. Also, make sure that it is saved in UTF-8 format, and not ANSI, using the File/Save As menu item in Notepad.

Thanks! It's running now. I was saving in ANSI (didn't even notice there was a choice).

Thanks again RoderickGI and rammingSpeed.
Title: Re: Is anyone running XMLTVDB to enhance xmltv.xml data?
Post by: muzicman0 on September 03, 2015, 02:50:18 pm
Sorry, I see you have this resolved, but I am also using it.  I ended up creating a batch file that would kill the application (by deleting the file mentioned in the folder the .jar file is running in), did a loopback ping (just to delay the net command), and then runs the .jar file. 

I found that if I just left it running all the time, it would eventually close (possible crash?) after a couple weeks.  Now that I am running a fresh copy every night, it works fine.  I just created a windows task to call the batch file.  Then I run MC2XML a while later, and then kill the .jar file again.

Been running like this since July I think, and it works great.
Title: Re: Is anyone running XMLTVDB to enhance xmltv.xml data?
Post by: CountryBumkin on September 03, 2015, 03:36:21 pm
Thanks for the additional info. I'll keep an eye on it and watch to see if it crashes after a couple of days or week.
I currently have it setup in Task Manager to run on startup (then it stays running, monitoring the mc2xml file for a change). It could probably be set up in task manager to run the mc2xml.exe file first (per schedule) then run the jar file, then shut down. For now, I'll just leave it running and see how it does.

Musicman are you in USA, and are you OTA or Cable card?
Title: Re: Is anyone running XMLTVDB to enhance xmltv.xml data?
Post by: muzicman0 on September 03, 2015, 03:38:11 pm
I am in the USA.  I have 4 tuners that are OTA, and 6 tuners that are cable card.  I use stacking to match up the OTA with the cablecard, and set it to prefer the OTA if available (on all but one channel - ABC - where the OTA has some minor quality issues).
Title: Re: Is anyone running XMLTVDB to enhance xmltv.xml data?
Post by: RoderickGI on September 03, 2015, 06:17:01 pm
Good to hear you got it working. The UTF-8 issue was listed as an issue on the XMLTVDB source site, and there is another version that fixes that requirement attached to a comment. I didn't download it but it might be a good idea, to avoid future issues, particularly as that source site is going to be archived next year. The program and fix may not be available after that happens. See https://code.google.com/p/xmltvdb/issues/detail?id=1
EDIT: I have now downloaded the fix, but not tested it yet.

You highlighted the wrong line, and missed the most important one!

Yaobing talked about importing "onscreen" Season and Episode format data, but I'm not sure if it was implemented, as it can be in any format. It looks like the XMLTVDB author has chosen the logical format though. Anyway, he has included the xmltv_ns format, which MC does read correctly.

    <episode-num system="xmltv_ns">7.18.0/0</episode-num>
    <episode-num system="onscreen">S08E19</episode-num>


I ended up creating a batch file <snip> did a loopback ping (just to delay the net command), <snip>

A TIMEOUT command would be more reliable than using ping. See http://www.robvanderwoude.com/wait.php
But either works I guess.
Title: Re: Is anyone running XMLTVDB to enhance xmltv.xml data? [solved]
Post by: CountryBumkin on September 05, 2015, 06:37:31 am
I have the fix download now. Thanks.

I don't know which line is used by MC (Onscreen or xmltv_ns) but I am getting Season Number now when XMLTVDB runs.

I'm having some trouble getting XMLTVDB in Task Scheduler to reliably run. Sometimes the XMLTVDB program doesn't want to run when MC2XML changes. Sometimes if I run the program manually the Command-prompt window will display showing XMLTV running, but XMLTVDB only runs a few lines of code then sits idle. Maybe I'm not waiting long enough - but seems like its frozen so I just close the window.

Is there a way to force the DOS/Command-Prompt window to open/show so I can see what the program is doing? I have Java Console set to "show" and the checkbox for "hidden" in Task Scheduler is unchecked.
Title: Re: Is anyone running XMLTVDB to enhance xmltv.xml data? [solved]
Post by: muzicman0 on September 05, 2015, 11:27:49 am
what command are you using to launch the .jar file?
Title: Re: Is anyone running XMLTVDB to enhance xmltv.xml data? [solved]
Post by: CountryBumkin on September 05, 2015, 11:46:52 am
In Task Scheduler, the Action is this: "C:\Program Files (x86)\Java\jre1.8.0_51\bin\java.exe" -jar "C:\mc2xml\xmltvdb.jar"

The xmltvdb folder is a subfolder of mc2xml in the root of C:. The XMLTVDB program ran on 9/3, but mc2xml has updated twice since then and XMLTVDB has not run again.
I've been checking security permissions and making sure the virus scanner was not blocking anything.

I can edit the mc2xml file and then "save as" to change the file date and XMLTVDB Logs show that the change was noticed.
Title: Re: Is anyone running XMLTVDB to enhance xmltv.xml data? [Almost solved]
Post by: muzicman0 on September 05, 2015, 06:36:02 pm
I'm using some batch files.

This one is StartXMLTVDB.bat
Code: [Select]
del "C:\Users\Steven\Documents\XMLTV\DELETE_ME_TO_STOP_XMLTVDB_LISTENER"
ping 127.0.0.1
ping 127.0.0.1
java -jar "C:\Users\Steven\Documents\XMLTVDB v1.0.1\XMLTVDB.jar"

This one is StopXMLTVDB.bat
Code: [Select]
del "C:\Users\Steven\Documents\XMLTV\DELETE_ME_TO_STOP_XMLTVDB_LISTENER"

at 12:50 AM every morning I run the StopXMLTVDB batch file.  Just in case it is running.  This may not need to be done, but I figured it was safer.
At 1:00 AM every morning, I run mc2xml to get the new xmltv file.
At 4:00 AM MC20 imports the new enhanced XML file that was created after the 1:00 AM batch file was run.
At 5:00 AM, I run the StopXMLTVDB.bat file.
Title: Re: Is anyone running XMLTVDB to enhance xmltv.xml data? [Almost solved]
Post by: RoderickGI on September 05, 2015, 08:19:00 pm
The XMLTVDB logs are pretty good. You should be able to see if it looked at the new files, and whether it tried to enhance the XMLTV file. The current logs are in the root directory of XMLTVDB, while the previous logs are in the "logs" directory.

Also, as shown by Musicman0's timing, XMLTVDB has to be running before mc2xml creates the new XMLTV file in the watched directory.

I found that just changing the "Date modified" of a file in the directory being watched wasn't enough to trigger XMLTVDB. The input file had to be created in the directory XMLTVDB was watching, and it didn't matter if it was exactly the same file that had been processed before, with the same "Date modified". I just deleted the previous file and copied a new version of it from the original I was using for testing. I assume that mc2xml deletes the previous file at some stage, then saves the new one, which should be sufficient to get it to run.

I couldn't get the Java Console to display either. According to the documents at Oracle "It is available for applets running with Java Plug-in and applications running with Java Web Start." So it isn't available for XMLTVDB. I looked at Tracing and Logging, but they don't provide much info either. So stick with the XMLTVDB logs.

I am going to keep XMLTVDB in my kitbag of tools, now that I have had a play with it and know it works. I don't need it while I am using EPG Collector, but you never know what will be needed in the future.  8)
Title: Re: Is anyone running XMLTVDB to enhance xmltv.xml data? [Almost solved]
Post by: CountryBumkin on September 06, 2015, 06:22:22 am
Thanks for the tips.

XMLTVDB ran okay last night.

I was unable to manually trigger it to run,  and your information about triggering a test run (delete first, then copy same version back to folder) is very useful.

Some times I get a Command Window when I "Run" mc2xml or XMLTVDB from the Task Scheduler. I'm not sure why the Command Window displays sometimes and other times it does not. I don't know if the Java Console and the Command Window I get are the same thing (it may be that the Java Console has never shown itself).

I tried EPG Collector on another computer. The results where not very good (North America terrestrial, OTA).  A number of stations had no EPG data. So in my case, mc2xml + XMLTVDB seems to be the best solution.

UPDATE: XMLTVDB has run successfully two nights now. I'll keep an eye on it but I think it is now setup correctly. Thanks to everyone for the help.
Title: Re: Is anyone running XMLTVDB to enhance xmltv.xml data? [Almost solved]
Post by: muzicman0 on September 06, 2015, 09:45:21 am
at 12:50 AM every morning I run the StopXMLTVDB batch file.  Just in case it is running.  This may not need to be done, but I figured it was safer.
At 1:00 AM every morning, I run mc2xml to get the new xmltv file.
At 4:00 AM MC20 imports the new enhanced XML file that was created after the 1:00 AM batch file was run.
At 5:00 AM, I run the StopXMLTVDB.bat file.
Realized I made a stupid mistake...at 12:50 AM I actually run the StartXMLTVDB.bat file.  This batch file will kill the XMLTVDB jar file if it is still running (although it shouldn't be running, it is a 'just in case' scenario).

Sorry about the confusion!
Title: Re: using XMLTVDB to enhance xmltv.xml data - How to get it to work?
Post by: CountryBumkin on October 21, 2015, 08:31:55 am
I'm reopening this old thread because I stuck trying to get XMTVDB program to finish running on my laptop (testing) computer.

I run mc2xml (stand-alone version 1.3) on my Server computer and then run the program XMLTVDB to provide Season and Episode Numbers for my recorded TV Shows.

The XMLTVDB program reads the mc2xml created file "xmltv.xml", then runs and creates a new file called "xmltv.enhanced.xm"l which is then imported by JRiver program.
No problems with either program on my Server computer. It runs everyday.

However, on my laptop computer, I have the exact same setup (same programs, folders/files locations, etc.) yet the XMLTVDB program does not complete. It starts running (I can watch it in the Command window) but seems to get stuck a few minutes into the process. it never completes, and doesn't create the "xmltv.enhanced.xml" file.

I have tried changing the settings "HTTP_Connection_Timeout", "HTTP_Read_Request_timeout", HTTP_Request_retries", and "Wait_seconds_between_retries" in the XMLTVDB properties file, but nothing helps. I don't know what the best combination for these settings are - and maybe the issue isn't involed with these settings.

Does anyone have any suggestions or ideas what I should look at to solve this?
Title: Re: using XMLTVDB to enhance xmltv.xml data - How to get it to work?
Post by: muzicman0 on October 21, 2015, 08:49:52 am
What version of Java is installed?
Title: Re: using XMLTVDB to enhance xmltv.xml data - How to get it to work?
Post by: CountryBumkin on October 21, 2015, 10:32:58 am
1.8.0_60 (the latest).

I just manually ran it (I opened a Command Window and ran "Java -jar xmltvdb.jar", then I moved xmltv.xml out of the monitored folder and back in again to start the process, and it worked). I got the xmltv.enhanced.xml file. So the program works, manually.

I'll need to wait until tomorrow now to see if it works/runs automatically when the next xmltv.xml (from mc2xml program) generates.

The settings I'm using now are:

Max Series lookup = 5 (default)
Max Wait_Hours_For_New_xmltv = 49 (default)
HTTP_Connect_Timeout = 60 (default is 20)
HTTP_Read_Timeout = 60 (default is 60)
HTTP_Request_Retries = 5 (default is 3)
HTTP_Time_Between_Retries = 10 (default is 15).

I don't know if these latest settings changes help or if it was just coincidental.
Title: Re: using XMLTVDB to enhance xmltv.xml data - How to get it to work?
Post by: muzicman0 on October 21, 2015, 12:11:47 pm
are you running the command "Java -jar xmltvdb.jar" in the folder where XMLTVDB.jar is located?  If not, you may need to specify the path to the jar file.  Put the path in quotes, but not the Java -jar part.  So it would be something like:

java -jar "C:\path to\XMLTVDB.jar"

if there are no spaces in the path then you probably won't need the quotes.
Title: Re: using XMLTVDB to enhance xmltv.xml data - How to get it to work?
Post by: CountryBumkin on October 21, 2015, 02:02:04 pm
When I run the program manually in "Command Prompt" I only change to the directory to where XMLTVDB is located (i.e.in Command prompt window I enter "cd c:\mc2xml/xmltvdb") before entering the command just as I listed it above. No quotes used. The path to the Java program is already stored in Windows "Environment Variables>System Variables". It runs okay manually (at least it did earlier today).

But the program (XMLTVDB) should run automatically whenever it detects a change to the "xmltv.xml" file. This is the part that I think is having trouble. It doesn't seem to want to run on its own. I can see in the Debug log of XMLTVDB that it detects the change in the xmltv.xml file but it doesn't seem to do anything after that.

As I mentioned before, this does work properly on my Server computer using the same XMLTVDB properties. I wonder if this is a Windows permissions issue.
I'm on Win10 x32 and was using the Windows Logon Account instead of Local Account for a little while - but now I'm logging in with local account (as the Windows Account cause networking permission issues and other problems). Maybe there is some left over garbage from the earlier setup.
Title: Re: using XMLTVDB to enhance xmltv.xml data - How to get it to work?
Post by: muzicman0 on October 21, 2015, 02:08:22 pm
I think I may be confused...you say the .jar file should run automatically when the xml file is updated...this isn't true (unless you have something else going on)...the .jar file has to be started, then once the .xml file changes, the processing will start, but the .jar file won't start unless you tell it to.
Title: Re: using XMLTVDB to enhance xmltv.xml data - How to get it to work?
Post by: CountryBumkin on October 21, 2015, 03:57:29 pm
The XMLTVDB program is always running in the background. Providing you start it manually once or use Task Scheduler to start it whenever your computer starts. I have confirmed that it is running (it just doesn't seem to work consistently on my laptop. It does work consistently on my Server computer without any action from me).

From XMLTVDB website https://code.google.com/p/xmltvdb/wiki/UserGuide

Execution¶

To run this program, execute:
"path\to\java.exe" -jar "path\to\XMLTVDB.jar"

This only has to be done once (recommended to schedule it to be run when the computer starts) and the program will listen infinitely for new XMLTV files. You can configure the program by editing the XMLTVDB.properties file that is next to the .jar.

To stop this program, delete the file named DELETE_ME_TO_STOP_XMLTVDB_LISTENER in the listening folder.  
Title: Re: using XMLTVDB to enhance xmltv.xml data - How to get it to work?
Post by: muzicman0 on October 21, 2015, 04:00:54 pm
OK, that makes sense, I thought you were saying that the .jar file would start when a change was done.  If the .jar file is already running, then yes, once a change is made, then it will process the file.

I am not sure why it isn't working on your laptop.  I run it on my server as well, with no issues. 

I would check all of your tasks, and make sure they are created correctly, and also check the times everything runs, perhaps you are starting the .jar file after you get the xml file...
Title: Re: using XMLTVDB to enhance xmltv.xml data - How to get it to work?
Post by: CountryBumkin on November 04, 2015, 03:50:29 pm
UPDATE:

I found that on the laptop, I need to leave the "Command Window" open after the XMLTVDB program runs. If I leave it open, the next day the XMLTVDB will run again without any manual intervention (triggered by the "xmltv.xml" file update) and it works 100% this way.

Oddly, on my other computer (my home Server) I don't need to leave the Command Window open. XMLTVDB just does its thing every time.

So at least it is working on the laptop now. It's been a week or so now and XMLTVDB has run every day (with Command Window open but minimized to taskbar). I can live with that.
Title: Re: using XMLTVDB to enhance xmltv.xml data - How to get it to work?
Post by: muzicman0 on November 04, 2015, 04:03:24 pm
I was just thinking I needed to come back and post in this thread.  I just discovered today that the new version of MC2XML has a Schedules Direct JSON option.  it appears that this pulls in Season and Episode info, so based on my tests, I will no longer be using XMLTVDB.  Of course if you aren't using SD, then XMLTVDB is still a good option.
Title: Re: using XMLTVDB to enhance xmltv.xml data - How to get it to work?
Post by: CountryBumkin on November 04, 2015, 05:00:23 pm
I have a SD subscription.

How does the JSON part work? Does JRiver need to add some programming/code to use this - or do you just get the latest version of Mc2xml and run it as stand-alone?
Title: Re: using XMLTVDB to enhance xmltv.xml data - How to get it to work?
Post by: muzicman0 on November 04, 2015, 05:12:44 pm
Yeah, just get the latest version (v1.5), and run it, and select the Schedules Direct JSON version...it will create an XML file just like before, the only difference is that you will no longer need to process is with XMLTVDB.
Title: Re: using XMLTVDB to enhance xmltv.xml data - How to get it to work?
Post by: RoderickGI on November 04, 2015, 06:44:01 pm
Have you tested it with PercData, which would be the more appropriate source?
Title: Re: using XMLTVDB to enhance xmltv.xml data - How to get it to work?
Post by: muzicman0 on November 04, 2015, 06:56:36 pm
this is a schedules direct feature, and not a general feature...SD includes the season and episode info in the json API, so now the mc2xml application can get it and include it in the xmltv file it generates.
Title: Re: using XMLTVDB to enhance xmltv.xml data - How to get it to work?
Post by: RoderickGI on November 04, 2015, 07:22:15 pm
Yes, I understand. But PercData is just the commercial arm of Schedules Direct, and is intended for use with applications such as MC.

So the same data is available, from the same source, just with different licencing. I was just wondering if mc2xml has the option to select PercData and the JSON API. I think the PercData integration in MC uses mc2xml to download the data. But I'm not in the USA so haven't used it. . .

I just looked at the mc2xml site and there is no mention of PercData, so maybe not. Then again, didn't someone report getting Season and Episode data via PercData and while back? If so, no need for the JSON API, or it is already being used. But Yaobing said no, he is using a SOAP call to initialise the connection, and then . . . something else to get the data.

Ah, here: http://yabb.jriver.com/interact/index.php?topic=100757.msg699403#msg699403
Title: Re: using XMLTVDB to enhance xmltv.xml data - How to get it to work?
Post by: CountryBumkin on November 05, 2015, 07:25:43 am
I tested PercData with their free 7-day trial, and the data was the same, or as good, as I was getting with Schedules Direct. As you know, SD is not supposed to be used with JRiver according to the license, so a JRiver user would need to use PercData to be correct/legal.

However, the EPG data from PercData still did not contain all the correct Season/Episode numbers for my TV recordings. Some where correct and some where wrong. For example, I recorded two episodes of "the Big Bang Theory" back-to-back on the same channel, and one had correct Season "number" and the other had Season "year". That is why I went back to using XMLTVDB.

Now with this news of mc2xml ver1.5 and JSON, I will look into testing that.

UPDATE: I installed mc2xml (version 1.5) today and selected the "Schedules Direct JSON" data source in the popup. Then I ran the program and downloaded a new xmltv.xml file. 

Looking at this file (for the series: Big Bang Theory, episode: The Spock Resonance) I see there are two entries now related to "episode number", there is "episode-num system= dd-progid > EP00931182.021" and there is a "episode-num system="xmltv_ns" >8.6.  NOTE: that this show is Season 9 episode 7 (I know that the season and episode counting starts at "0" for season or episode #1, hopefully MC does too).
I suppose I will find out tomorrow if the recording scheduled for tonight will have correct Season and Episode numbers.

Thanks for bringing this to my attention!
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: muzicman0 on November 05, 2015, 11:43:53 am
Yeah, MC interprets the 8.6 in the xmltv_ns as season 9 episode 7.
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: RoderickGI on November 05, 2015, 03:58:06 pm
It is strange that you got bad Season data from PercData as it is exactly the same data as SD. Perhaps it was a one off glitch, or at least the sort of thing could be fixed by telling PercData about it.

In the post I linked to Yaobing said he would have to figure out if he could use the JSON calls to get more metadata, such as Season and Episode numbers. Elsewhere he had said he was surprised that you were getting any Season and Episode data, because either he wasn't requesting it or it wasn't supposed to be in there. So, MC may need to use JSON calls to get quality metadata, and that may be the fix.

Anyway, I just raised PercData because that is the correct source to use. If I evangelise for those guys, maybe they will add Australian data to their system!  ;D :D
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: CountryBumkin on November 06, 2015, 05:00:19 am
It is strange that you got bad Season data from PercData as it is exactly the same data as SD. Perhaps it was a one off glitch, or at least the sort of thing could be fixed by telling PercData about it.

In the post I linked to Yaobing said he would have to figure out if he could use the JSON calls to get more metadata, such as Season and Episode numbers. Elsewhere he had said he was surprised that you were getting any Season and Episode data, because either he wasn't requesting it or it wasn't supposed to be in there. So, MC may need to use JSON calls to get quality metadata, and that may be the fix.

Anyway, I just raised PercData because that is the correct source to use. If I evangelise for those guys, maybe they will add Australian data to their system!  ;D :D


I get bad Season and Episode "Numbers" using Schedules Direct data too. The problem is that a lot of the Season "Numbers" come in as Season "Year". That's why I started using XMLTVDB.

Hopefully, the new mc2xml (ver 1.5) using  the new Schedules Direct "JSON version" will - or I'll just keep using mc2xml with XMLTVBD.
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: imeric on January 17, 2016, 09:09:03 pm
I'm getting the following error trying to use xmltvdb:

The reference to entity "w" must end with the ';' delimiter

does anybody know why?
thx
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: muzicman0 on January 17, 2016, 11:00:01 pm
Did you change the config file?  Perhaps something in there?
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: RoderickGI on January 17, 2016, 11:12:52 pm
http://yabb.jriver.com/interact/index.php?topic=99519.msg712202#msg712202
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: imeric on January 18, 2016, 08:05:22 am
I just tried this morning and I still get the following error

Code: [Select]
01/18/2016 08:55:22 AM INFO    Logger has been initialized after a reset
01/18/2016 08:55:22 AM NOTICE  Processing: M:\mc2xml_OTA\xmltv.xml
01/18/2016 08:55:22 AM ERROR   Error processing xml file: org.jdom.input.JDOMParseException: Error on line 304 of document file:/M:/XMLTVDB%20v1.0.1/xmltvs/xmltv.xml: The reference to entity "w" must end with the ';' delimiter.
                                   org.jdom.input.JDOMParseException: Error on line 304 of document file:/M:/XMLTVDB%20v1.0.1/xmltvs/xmltv.xml: The reference to entity "w" must end with the ';' delimiter.
                                   org.jdom.input.SAXBuilder.build(SAXBuilder.java:530)
                                   org.jdom.input.SAXBuilder.build(SAXBuilder.java:905)
                                   org.jdom.input.SAXBuilder.build(SAXBuilder.java:884)
                                   xmltvdb.xmltvdb.processXMLTVFile(xmltvdb.java:301)
                                   xmltvdb.xmltvdb.access$100(xmltvdb.java:67)
                                   xmltvdb.xmltvdb$1.onEvent(xmltvdb.java:178)
                                   btv.folderwatcher.FolderWatcher.run(FolderWatchers.java:79)
                                   java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
                                   java.util.concurrent.FutureTask.run(Unknown Source)
                                   java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
                                   java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
                                   java.lang.Thread.run(Unknown Source)
                                   
01/18/2016 08:55:22 AM NOTICE  --------------------------------------------DONE PROCESSING XMLTV FILE--------------------------------------------
01/18/2016 08:55:22 AM DEBUG   Not sending e-mail notification because it is disabled in config file.
01/18/2016 08:55:22 AM DEBUG   LISTENER: Skipping Event ENTRY_MODIFY. File is: xmltv.xml

And you will find my properties file attached.

Help Please thx.
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: muzicman0 on January 18, 2016, 09:01:59 am
It looks like it is an error in the actual XML file.  Have you been trying on the same XML file, or have you been downloading a new one?  I would try to get a new XML file to process.

Also, if I remember, it requires Java 1.6 or higher, what version of Java is on your system?
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: CountryBumkin on January 18, 2016, 09:19:18 am
It's Java 7 or higher, so this should not be a problem on any current Windows OS.

Your XMLTVDB properties file looks fine.

Once you have the XMLTVDB program installed, you can manually run it by opening a Command Prompt Window, then type cd C:\mc2xml (or wherever it is located) Java -Jar xmltvdb.jar
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: muzicman0 on January 18, 2016, 09:42:41 am
Java is not installed by default in Windows, so depending on when Java was installed, it may be an issue.  it is certainly worth checking.
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: imeric on January 18, 2016, 10:26:55 am
Installed latest Java update (8.1 rel66)
It also uninstalled 8.1 rel51
rebooted.  Tried again and it worked!

It's doing its thing now will post back on results.  A good old reboot always seems to fix things :)
ANyway to add other languages in the search criteria in the properties file?

thx
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: imeric on January 18, 2016, 10:41:39 am
Looks like TVDB is offline again.? Started working but then stopped. DOn'T get anything either form their web page..
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: RoderickGI on January 18, 2016, 03:20:37 pm
It doesn't look like http://thetvdb.com/ was planned to be down. But they may still be having issues after the recent maintenance.

Perhaps check the site before running your next test, so you know it should work? Assuming, of course, that your internet connection is reliable.
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: imeric on January 18, 2016, 04:32:58 pm
Fiber to the Home.  Can't beat that :)

Anyone else experiencing issues? Have you tried running it RoderickGI?
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: RoderickGI on January 18, 2016, 05:02:55 pm
Fiber to the Home.  Can't beat that :)

Anyone else experiencing issues? Have you tried running it RoderickGI?

I dream of fibre. I have 30Mbps download now, and could get 100Mbps for some more $$$s. But I only have 1Mbps upload, and the best I can buy is 2 to 2.5Mbps. Bummer.

I ran XMLTVDB successfully some time back. See my posts early in this thread.

I would use XMLTVDB myself, but EPG Collector already does the same thing for me.
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: imeric on January 18, 2016, 10:49:26 pm
Yeah I saw your other posts...Mmm EPG collector..Should I look at that...Oh boy..Getting late, I'm spending WAY TOO much time on this..This is kinda why I'm always reluctent to re-try MC for TV...But now I feel I've gone too far :):)....Might be too late to go back!!
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: muzicman0 on January 18, 2016, 11:10:46 pm
looks like thetvdb is back up...or at least it just loaded for me.
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: RoderickGI on January 19, 2016, 12:17:18 am
EPG Collector collects data from the broadcast stream though. Almost out only option in Australia. But I know that isn't very good in most parts of America. You are in Canada though aren't you? I don't know how good the OTA EPG is there.

That is why Zap2it (Gracenote), Microsoft (Rovi), PercData(Gracenote), (and Schedules Direct (Gracenote) if you are naughty) are better options in America.
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: RoderickGI on January 19, 2016, 12:19:23 am
looks like thetvdb is back up...or at least it just loaded for me.

It has been up and down all day. If it is up, do a test search at the episode level for "The Good Wife" Perhaps S07E12, "Tracks". Even when the site was up, I couldn't get that search to work.
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: CountryBumkin on January 19, 2016, 07:02:30 am
It's Tuesday morning now, I checked my EPG data update scheduled to run at 6AM and it worked. I looked up "The Good Wife S07E12" and had no problem with that either. So perhaps whatever was going on with TheTVdb.com is over now.

@imeric, You said a few posts back that you got XMLTVDB working. Is it good now - or are you still working on issues?

BTW - I took a look at EPG Collector a while back, but never got it working - then everything came together with mc2xml>XMLTVDB (and then with "mc2xml>Schedules Direct JSON") so I never went further with Collector. If you do get it setup, let us know how the data compares with the XMLTVDB data.
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: imeric on January 19, 2016, 07:53:39 am
It was working yesterday after updating Java and a reboot but I was getting timeout errors while it was fetching data throughout the day.
Now back to square one I just tried it again and I get this stupid error again:  

Code: [Select]
01/19/2016 08:49:18 AM ERROR   Error processing xml file: org.jdom.input.JDOMParseException: Error on line 305 of document file:/M:/XMLTVDB%20v1.0.1/xmltvs/xmltv.xml: The reference to entity "w" must end with the ';' delimiter.
Help Please!!
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: muzicman0 on January 19, 2016, 08:54:58 am
Can you attach the XMLTV file you are trying to process?
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: CountryBumkin on January 19, 2016, 09:28:59 am
I was having problems/errors running XMLTVDB on my laptop (Wi-Fi) so I modified the timing settings in the XMLTVDB properties files to this:

The changes I made are:

Max Series lookup = 5 (default)
Max Wait_Hours_For_New_xmltv = 49 (default)
HTTP_Connect_Timeout = 60 (default is 20)
HTTP_Read_Timeout = 60 (default is 60)
HTTP_Request_Retries = 5 (default is 3)
HTTP_Time_Between_Retries = 10 (default is 15).

I don't know if it was coincidental or if the settings above fixed it, but it started working after that change.
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: imeric on January 19, 2016, 10:54:35 am
I made the following changes thx.

I know it's not the xmltv file as the same file that was giving the error worked yesterday after the Java update and a reboot.

I'll try the reboot again.  + The xmltv file is 15 MB.... I could trim it a little and send it later but I'll try the reboot first.

Any other idea what might cause this? I re-attached my properties file.

Thx folks 
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: muzicman0 on January 19, 2016, 11:00:40 am
If you have a drop box, you could give me a link.  I can test it as soon as I have a copy...just to be sure.  The error you are getting is specifically referencing the XMLTV file, so I think it is worth checking on a known working system.
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: imeric on January 19, 2016, 11:59:35 am
Here it is.
Thx for checking!
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: muzicman0 on January 19, 2016, 02:51:22 pm
I get the same error on that XMLTV file.  I am looking deeper at it now.
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: muzicman0 on January 19, 2016, 03:40:47 pm
It appears to have a problem with lines that start with <icon src=.  I removed line 305, and re-ran the jar file...then it had a problem with another line...did this about 5 times...just kept jumping to a new line that started with the icon key. 

I then downloaded from my SD account and ran, and it ran just fine.  I checked, and the SD XMLTV file does not have any <icon src= lines.

At this point, I don't know what else to try.  Sorry.

If you had a way to filter out those lines, it may work.  I tried to create a powershell script that would do just that, but interestingly enough, it failed on the same line.
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: RoderickGI on January 19, 2016, 04:24:56 pm
Quote
2015-08-11: MS has added ampersands to the program images. Send an email if your PVR software has any problems importing the xmltv file (due to the icon tag with unencoded &amp;) and I will reply with an update that encodes/fixes it (a quick fix if your PVR doesn't use images: sed "/<icon/d" xmltv.xml > xmltv.out )

From http://mc2xml.awardspace.info/ Service information(9);
Reference https://en.wikipedia.org/wiki/Sed#Other_sed_commands

Might be a bit harder in Windows to delete all the icon lines.
This, also from http://mc2xml.awardspace.info/  Service information(9);
Quote
Microsoft media center Win7+ service: If you have used Windows 7 or 8 you might have noticed more program information in the guide. It uses a completely new TV data service that seems to support everything that the microsoft legacy service supported*, but adds full ATSC/subchannel support ("OTA") for the US & Canada, plus listings include writer/producer credits, program images  , as well as other miscellaneous program details. (note: Microsoft might have a North American focus?)

But, why isn't everyone else having the same problem? Is there a setting in the mc2xml setup in MC to ask for images and/or channel icons? Of course, it may just be that certain broadcast area get the images, and others don't.
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: CountryBumkin on January 19, 2016, 05:28:46 pm
@imeric - are you in Canada? If so, maybe that's the difference.
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: imeric on January 20, 2016, 08:02:10 am
I'M in Canada. But I get the same error with an xmltv file with a ZIP code close to the border.  So Everyone using the XMLTV.xml file coming from Rovi will get the same error right?
I'm also using MC2XML v1.4 (same as what's in MC21.0.37)

Any idea on what to do next or it's a dead end with Rovi and XMLTVDB?

Thank you all for your help on this. Much appreciated.
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: CountryBumkin on January 20, 2016, 09:02:36 am
I don't use XMLTVDB anymore. Since I have a subscription to Schedules Direct, and SD now has the JSON data available, there is no need for me to run the extra step/program (XMLTVDB).

I can run mc2xml built-in to MC to generate a "xmltv.xml" file and I get no error.  Give me your Zip code and I'll try for you (are you using "OTA broadcast" or using "cable" or something else?

To be clear, is the error on the xmltv.xml file (generated from mc2xml), or are you getting it running the XMLTVDB program (which would then create the file "xmltv.enhanced.xml"?
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: imeric on January 20, 2016, 12:34:31 pm
Hi Countrybumkin. 12919 is the Code with OTA.

Are you using the updated jar or just the one in the 1.09.1 zip file on the XMLTVDB website?

The error is coming from the error log in XMLTVDB. It's strange as I had it working ONCE after the java update and reboot but was not able to re-running it without this error...
Well it ran but never produced the enhanced file. At least it went past the error above...

Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: CountryBumkin on January 20, 2016, 12:55:37 pm
I'm using XMLTVDB version 1.0.1 before that it was 1.0
I've never seen a 1.09.1

Are you getting your file here: https://code.google.com/p/xmltvdb/downloads

UPDATE: I just ran mc2xml with your zip code and then XMLTVDB (ver 1.0.1) and it running fine.
If your not using 1.0.1 upgrade to that first, if your still having trouble, I'll go through step by step with you.
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: imeric on January 20, 2016, 02:22:25 pm
Yes I am but I saw an update for 2014 for the jar file maybe that's the issue...
I'll try on my other PCs and post back.  WHich version of Java are you using?
again, many thx for the help with this
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: CountryBumkin on January 20, 2016, 02:28:24 pm
There is an error in your xmltvdb properties file. You have this
#The folder to listen to. Any XMLTV files put in this folder will be enhanced and moved to the OUTPUT_XML_FILE location
WATCH_FOLDER=M:\mc2xml_OTA
#The final destination (including file name) of the enhanced XMLTV file.
OUTPUT_XML_FILE=M:\mc2xml_OTA

You should have this:
#The folder to listen to. Any XMLTV files put in this folder will be enhanced and moved to the OUTPUT_XML_FILE location
WATCH_FOLDER=M:\mc2xml_OTA
#The final destination (including file name) of the enhanced XMLTV file.
OUTPUT_XML_FILE=M:\mc2xml_OTA\xmltv.enhanced.xml

I have Java 1.8.0_60

Also you have this #Pipe | seperated list of XMLTV source-info-name attributes that identify the guide data as from Zap2It
ZAP2IT_SOURCE_INFO_NAMES=Zap2it|Schedules Direct|Rovi

But "Rovi" is not in the code (at least not in 1.0.1, and that version works fine). I don't know if adding "Rovi" matters or not - probably not. I think once you add the "output file name" it will work.
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: imeric on January 20, 2016, 02:57:06 pm
Yes I changed it a few minutes ago and that would explain why it didn't output the enhanced file the ONE time I was able to run it....

 I also updated to the latest Java release (71) and still not working.

I will try on a few other PCs to see what happens...If you have any other suggestions don't be shy :)!

Thx a lot for your help with this!
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: imeric on January 20, 2016, 02:58:59 pm
Oh and are you running Win10?  I also deactivated McAfee, put excusions for the jar file but it didn't change anything...
I have one WIn7 machine I'll try with..thx
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: CountryBumkin on January 20, 2016, 03:29:53 pm
You can try copying my file setup exactly if you want. It might make it easier to figure out why it runs on my computer and not yours.

1) create a new folder at C:\mc2xml
2) Put mc2xml.exe and mc2xml.dat files in this folder.
3) Run the mc2xml program. This will create the file "xmltv.xml".

4) copy (drag-drop) the XMLTVDB folder and its subfolders to C:\mc2xml, so you have "C:\mc2xml\xmltvdb v1.0.1"  (rename the "xmltvdb v1.0.1" to "xmltvdb" to make it easier to work with). There will be subfolders under xmltvdb (such as Lib, xmltvs).
5) Make the changes to the "xmltvdb.properties" file as described elsewhere (i.e. the watched folder is "C:\mc2xml" and output folder-file is "C:\mc2xml\xmltv.enhanced.xml").

6) test step 1: open a Command Window, type "cd C:\mc2xml\xmltvdb", then type "Java -jar xmltvdb.jar". This starts the XMLTVDb program to watch "xmltv.xml".
7) test step 2: leave the Command Window open, then use Windows File Explorer and move the "xmltv.xml" file out of the folder C:\mc2xml". Then move it back in to C:\mc2xml folder (that will trigger the XMLTVDB program to run the search).

Does this now work?

Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: imeric on January 20, 2016, 08:33:30 pm
Same error..But on the same PC...Let me try on another machine and see how it goes.
Has to be Java.  The ONE thing it worked was right after an update of Java....

Thank you CountryBumkin for all your help much appreciated and I really want this to work!!!
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: RoderickGI on January 20, 2016, 11:57:59 pm
The problem, as I said back here (http://yabb.jriver.com/interact/index.php?topic=99845.msg712597#msg712597), is the ampersand (&) in the "<icon src=..." line. The first of such lines in your sample file is line 305, which looks like this;

      <icon src="http://images-eds.xboxlive.com/image?url=cOC861nTwwD6fdaNNqSphYhIoPdt3SpAXZveWYNx1owgCAFwKkxN4BHYkOsXD3dIwKodgfbUrBAfnHsMhu._ZusWV2mP8djwAhyIBPnHHXxao5KMQaJ4PITjyW4RAAYcmQF8x27HBO4iEIK_N4PaOw--&w=267&h=150" />

The next line that has the problem is line 315. If I edit line 305 to add in the ampersand coding, shown below in red;

      <icon src="http://images-eds.xboxlive.com/image?url=cOC861nTwwD6fdaNNqSphYhIoPdt3SpAXZveWYNx1owgCAFwKkxN4BHYkOsXD3dIwKodgfbUrBAfnHsMhu._ZusWV2mP8djwAhyIBPnHHXxao5KMQaJ4PITjyW4RAAYcmQF8x27HBO4iEIK_N4PaOw--&amp;w=267&amp;h=150" />

Then XMLTVDB fails at line 315 instead of line 305. I tested with both a UTF-8 format files, and an ANSI formatted file using the version of XMLTVDB from this page (https://code.google.com/p/xmltvdb/issues/detail?id=1).

So, the problem is definitely the "&" in the "<icon src=..." lines. XMLTVDB needs the ampersand to be encoded.

It could be that no-one else is getting that line in their Rovi XMLTV files. CountryBumkin, perhaps if you ran your standard alone version of mc2xml that works with Rovi data for a few random US post codes, you could check to see if they have the "<icon src=..." lines.

Or it could be that no one else who gets those lines is using XMLTVDB to improve their data. After all, both mc2xml and MC cope with the format currently being used, which is using ampersands without encoding.

I don't think you are going to get anyone to fix XMLTVDB so it handles un-encoded ampersands. You may be able to find another piece of software that does the same thing. Another solution, is to ask Yaobing to contact the author of mc2xml and include a copy of mc2xml in MC that encodes the "&" to "&amp;", as long as that doesn't break MC functionality.

Over to you guys. I don't need this tool, so at this stage, I'm taking it no further.



Oh, one last point. As MC doesn't need Season and Episode numbers to look up TheTVDB any more, do you really need this tool now?
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: CountryBumkin on January 21, 2016, 07:59:32 am
I don't understand why imeric is having a problem running mc2xml and XMLTVDB when I ran the same program (using his postal code) and it works fine for me. If there are ampersands in imeric's XMLTVDB code they should also be in mine.
Perhaps it is his version of Java.
I'll try with the lastest Java version (71) next, just to know.

I didn't review the XMLTVDB log file imeric posted because it's RAR compressed and I don't have a program to open that.

@imeric, are you using XMLTVDB version 1.0.1 or the version you downloaded (1.09.1) that you said was newer? Perhaps that newer version is the problem.
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: imeric on January 21, 2016, 08:25:13 am
Hi CountryBumkin,
I tried with both version and it gives me the same result.  I still haven't tried on another PC I need to do that...

I really think it's either Java or Windows 10 or my Anti-Virus (Mcafee) but the AV is disabled...

I also run Dos as an admin and it leads to the same error.
Which version of Windows are you running?
WOuld you mind trying with the latest Java? here is what I have (see pic attached)
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: imeric on January 21, 2016, 08:41:43 am
Just tried on another PC but it has Win10 and the same AV...
Country Bumkin would you mind sharing your properties file I'll try with it and change my Directories only...

I will also try with a different PC with Win7 and Microsoft endpoint...Tonight..
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: JimH on January 21, 2016, 08:42:12 am
I really think it's either Java or Windows 10 or my Anti-Virus (Mcafee) but the AV is disabled...
Disabling an antivirus program is sometimes not enough.  Uninstalling it is the only way to eliminate it as the cause of a problem.
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: CountryBumkin on January 21, 2016, 08:43:28 am
Win10 x32 on my laptop (where I do my testing-playing around) and Win10 x64 at home (Server and Clients). I only have Windows Defender installed on laptop, at home I use Windows Defender and Malwarebytes.

This ampersand issue/error that is mentioned, where are you seeing that, in the XMLTVDB log file?
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: imeric on January 21, 2016, 08:47:09 am
Yep.

@Jim  Yeah I know, thx for the suggestion though...I'll try other things on another machine with a different AV and see what happens. (However it's my work PC and can't disable it we'll see how it goes)

As I was able to run it once after fiddling with Java versions I'm 99% convinced it has something to do with Java...Maybe this is what I should uninstall and reinstall I'll try that when I have a few minutes...

Thx all for the help!!!
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: CountryBumkin on January 21, 2016, 09:13:19 am
The error says the problem is in the xmltvdb\xmltvs\xmltv.xml file. Try copying my xmltv.xml file to your computer. Had to Zip it as MC says its too big to attach as is (32KB). Be sure to put it in the xmltvs subfolder, not the mc2xml folder.
I am including the xmltvdb.properties file too, but I don't think that is the problem as mine looks identical to yours.

Also why not include your file and I'll see if I get the error on my machine using your xmltv.xml file.


Interact is having a fit with my attachments. the "XMLTVDB.zip" is the "XMLTVDB.properties" files. Then Interact says my post with the new attachment is a "duplicate" so I have to start over. This is my fifth attempt now. It won't let me attach the xmltv.xml file (even compressed its too big).
Send me a PM and I'll e-mail the file to you.
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: imeric on January 21, 2016, 09:40:15 am

Oh, one last point. As MC doesn't need Season and Episode numbers to look up TheTVDB any more, do you really need this tool now?


Is this official? This will be in the next version?

I still would like to give XMLTVDB a shot..Just curious (and stubborn :))
CountryBumkin I'll send you my XMLTV
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: CountryBumkin on January 21, 2016, 09:49:50 am
I believe Roderick is referring to the discussion about having a blank "Season" or "Episode" number, and the Get Movie & TV Info tool will still work. Of course it's just nicer to get all the correct info in one shot (if there is an EPG source that provides it).
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: imeric on January 21, 2016, 09:52:05 am
If that's the case he needs to re-read my posts  as I've been repeating at least 5-10 times that I want this to be automated :):)

Countrybumkin another more simple option would be to use winrar as the compression rate is very good with those xml and you could grab it from my xmltv.xml file split in 2 in my earlier post. That's what RoderickGi used to test.

Let's see if you get the same thing as per his post:

The problem, as I said back here (http://yabb.jriver.com/interact/index.php?topic=99845.msg712597#msg712597), is the ampersand (&) in the "<icon src=..." line. The first of such lines in your sample file is line 305, which looks like this;

      <icon src="http://images-eds.xboxlive.com/image?url=cOC861nTwwD6fdaNNqSphYhIoPdt3SpAXZveWYNx1owgCAFwKkxN4BHYkOsXD3dIwKodgfbUrBAfnHsMhu._ZusWV2mP8djwAhyIBPnHHXxao5KMQaJ4PITjyW4RAAYcmQF8x27HBO4iEIK_N4PaOw--&w=267&h=150" />

The next line that has the problem is line 315. If I edit line 305 to add in the ampersand coding, shown below in red;

      <icon src="http://images-eds.xboxlive.com/image?url=cOC861nTwwD6fdaNNqSphYhIoPdt3SpAXZveWYNx1owgCAFwKkxN4BHYkOsXD3dIwKodgfbUrBAfnHsMhu._ZusWV2mP8djwAhyIBPnHHXxao5KMQaJ4PITjyW4RAAYcmQF8x27HBO4iEIK_N4PaOw--&amp;w=267&amp;h=150" />

Then XMLTVDB fails at line 315 instead of line 305. I tested with both a UTF-8 format files, and an ANSI formatted file using the version of XMLTVDB from this page (https://code.google.com/p/xmltvdb/issues/detail?id=1).

So, the problem is definitely the "&" in the "<icon src=..." lines. XMLTVDB needs the ampersand to be encoded.
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: CountryBumkin on January 21, 2016, 09:57:40 am
If that's the case he needs to re-read my posts  as I've been repeating at least 5-10 times that I want this to be automated :):)

I found the easiest way "for me" is just to pay for a Schedules Direct subscription, then I don't need to worry about running two programs (mc2xml and XMLTVDB), I just use mc2xmlver 1.5 and it provides me the Schedules Direct JSON data.

I have not bothered to compare the data from Zap2it to XMLTVDB to see which is more complete/better (although Zap2xml may be easier to run). I'm happy with SD data now that they added the JSON feature.
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: RoderickGI on January 21, 2016, 02:43:54 pm
Is this official? This will be in the next version?

It is official, and has been in the software December 14, and it works. The only disadvantage is that MC looks up the information after a recording is started, and so it is not in the EPG. But MC will find the metadata for a program based purely on the Series and Episode names. No Season or Episode numbers required.

Quote
Media Center 21.0.28 for Windows -- Available here
21.0.28 (12/14/2015)
5. Changed: TheTVDB lookup can look up tracks that don't have a season or episode number and will now look up just by the episode name.

I still would like to give XMLTVDB a shot..Just curious (and stubborn :))

I know. But still, XMLTVDB is no longer required, assuming that the new functionality works correctly. It would help JRiver if you tested that instead of working on XMLTVDB.  8)
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: imeric on January 21, 2016, 03:27:23 pm
All right I agree enough with XMLTVDB I looked at a few enhanced files and it doesn't seem like it's not gonna cut it anyway.

As per my several posts here: http://yabb.jriver.com/interact/index.php?topic=99519.msg712257#msg712257

I can guarantee you that this here:

Quote
21.0.28 (12/14/2015)
5. Changed: TheTVDB lookup can look up tracks that don't have a season or episode number and will now look up just by the episode name.
 
Is not working as it will always take S0EXX along with the episode name which means the search will fail.  This is what happens when I hit Get Movie & TV Info... I therefore assume it's doing the same with the auto-import.
If it was working then it would grab the info automatically with the auto-import after a show is recorded and it's not happening. Not with Rovi anyway...
My good wife snapshots and examples here: http://yabb.jriver.com/interact/index.php?topic=102045.msg712624#msg712624
AND I don't need the Season and or detailed info in the EPG if I get it after with the TVDB lookup on autoimport that's all I need.
thx
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: RoderickGI on January 22, 2016, 09:53:42 pm
I'm going to give this thread a little bit more of a poke, just for future reference and other potential users.

Are you using the updated jar or just the one in the 1.09.1 zip file on the XMLTVDB website?

First, as CountryBumkin said, I have never seen a version 1.09.1 of XMLTVDB. But I have seen many versions created by people other than the author, and sometimes they are packaged up with a version of Java from the time they are created. Maybe one of those versions is what you have got. Where did you get the version 1.09.1 from? You should be using version 1.0.1 from the official site CountryBumkin posted, https://code.google.com/p/xmltvdb/downloads/list.

As I was able to run it once after fiddling with Java versions I'm 99% convinced it has something to do with Java...

I am 100% convinced that it is nothing to do with the Java version you are using. I have run XMLTVDB on Java 8 Update 66 and 71, and it ran on earlier versions in the past. All worked fine. When you first ran XMLTVDB and it appeared to work, just that once, you didn't have the output file defined in the properties file, as noted later. I would be very sure that what happened is that XMLTVDB tried to run, checked the Output File setting or tried to create the output file, failed, and so stopped before attempting to process the XMLTV input file. It would have taken a while for XMLTVDB to give up on creating the output file, since it would be waiting for Windows to respond, and there were probably retries involved. Basically, I think you have to ignore that first attempted run as an anomaly, as that setting was wrong.

I don't understand why imeric is having a problem running mc2xml and XMLTVDB when I ran the same program (using his postal code) and it works fine for me. If there are ampersands in imeric's XMLTVDB code they should also be in mine.
UPDATE: I just ran mc2xml with your zip code and then XMLTVDB (ver 1.0.1) and it running fine.
If your not using 1.0.1 upgrade to that first, if your still having trouble, I'll go through step by step with you.

Did you check your output file to see if there were ampersands in it? If there were, did they have "amp;" after the ampersand, which XMLTVDB would then understand?

Are you sure you ran the same mc2xml version provided with MC as Imeric is running? You haven't replaced that version with the version 1.5 you have by any chance?

As I mentioned earlier, John Douglas, the author of zap2xml and mc2xml, has recognised a problem with reading un-encoded ampersands, and offered to provide a version that encodes them for programs that require it. He posted about that on 2015-08-11. Since then he released the Donateware version 1.5 of mc2xml, which may well have had the ampersand fix in it by default. So, if you used your version 1.5 of mc2xml to test Imeric's postcode, the XMLTV file mc2xml created may have had encoded ampersands in it, and XMLTVDB would therefore not have had a problem processing it.

By the way, mc2xml is up to version 1.6 now, and John seems to have removed the requirement to donate to get the version that works with Rovi data. I haven't tried to run it against Imeric's post code as I don't have a "Key" and "Secret" to run it independently, and I haven't tried to update the version in MC. In fact I would probably have to break my MC installation in order to test it, since my TV setup is completely different.

Finally, the change;
Quote
Media Center 21.0.28 for Windows -- Available here
21.0.28 (12/14/2015)
5. Changed: TheTVDB lookup can look up tracks that don't have a season or episode number and will now look up just by the episode name.
is working fine, but as you say, it gets stopped by the current practice of placing the year into the Season tag if it is blank, and that deleting the year from the Season tag means the lookup uses S0Exx, which will always fail.

As I have said elsewhere, the insertion of the year into the Season tag when it is empty needs to be stopped, but there also needs to be some more logic around the updating of the Season and Episode tags, because if the year isn't written to the Season tag, but the Episode number is provided by the EPG "onscreen" Season/Episode field, then the MC lookup will still fail because MC will use S0Exx. If the Episode tag wasn't updated when no EPG Season field was available, then the MC lookup by Series and Episode name would work.

Anyway, until that functionality is fixed, I have an idea on how it could be worked around. If an suitable expression could be written and added to each recording rule and the Auto-Import function for the location containing TV recordings, such that "if Season is <four numeric digits> OR <blank>, set Season and Episode tags to <blank>". With that in place, each recording made or imported into MC that did not have a non-year Season tag would have both tags deleted, and the MC lookup by Series and Episode name would work.

I might take a while to write such an expression, since I'm not up to speed on all of MC's expression language. It will also be a little difficult to test on my installation, without breaking things. But I could do it. However, if someone else wants to come up with the expression to be tested, great!
Title: Re: using XMLTVDB to enhance xmltv.xml data
Post by: astromo on January 23, 2016, 05:04:03 am
Yeah I saw your other posts...Mmm EPG collector..Should I look at that...Oh boy..Getting late, I'm spending WAY TOO much time on this..This is kinda why I'm always reluctent to re-try MC for TV...But now I feel I've gone too far :):)....Might be too late to go back!!

Stick with it is my tip.

Australian OTA EPG data is far from perfect and as Rod_GI points out, EPGC does what it can by enhancing data with a theTVDB scrape.

Even though MC can always be improved, the advantage of having all media action under one roof is what makes it work for me.