INTERACT FORUM

Please login or register.

Login with username, password and session length
Advanced search  
Pages: [1]   Go Down

Author Topic: [EPG] MC Kills my external EPG data collector  (Read 6470 times)

RoderickGI

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 8186
[EPG] MC Kills my external EPG data collector
« on: September 23, 2014, 04:05:13 am »

I have now run lots of tests with MC and my EPG provider, "EPG Collector". Here are my observations;

1. If I run the EPG Collector directly in Windows it always completes and generates an XMLTV file, ready to be imported into MC.
2. If I run EPG Collector manually from within MC using the Load Program Guide functionality, it always completes and generates an XMLTV file, which MC then imports successfully.
3. If I schedule EPG Collector to run within MC, it will start the run, but will terminate after about 4 to 15 minutes without completing.
   3a. The EPG Collector log recording just stops, with no indication as to why it stopped.
   3b. I can't find any reason in the Windows Event Viewer for the termination.
   3c. I can't make sense of the MC logs to find anything indicating why it killed the EPG Collector process. (Wait! I did!)
   3d. During initial tests EPG Collector was always terminated after exactly 10 minutes, but with later tests the time has varied.
   3e. It appears that the problem only occurs when EPG Collector is run "silently" by MC.

Some background.
I run the MC Load Program Guide process, using the XMLTV option and a third party application, "EPG Collector" as the external program which gathers EPG data.

EPG Collector takes about 2.5 minutes to gather the basic EPG data from the OTA broadcast transmission. This does not include Season and Episode numbers, so MC is unable to gather TV program information on import of the TV recordings I make. In order to get Season and Episode numbers into MC, I have turned on the EPG Collector Metadata Lookup functionality, which among other things matches the known Series and Episode names to Season and Episode numbers. This lookup process finds the data on the internet and takes an extra 15 minutes or so.

When I do not run the Metadata Lookup process EPG Collector completes successfully every time, because it finishes in a few minutes, I believe. Certainly I never noticed these failures to complete until the EPG Collector run time extended beyond a few minutes. The problems occurs in MC19 up to the latest, and in MC20.0.17. I upgraded to MC20 to get Yaobing's new Load Program Guide functionality.

So my question is; Why does MC kill the EPG Collector process before it completes?



I think I have found the answer myself. Well, the evidence anyway. I finally got a run done and saved a MC log straight after EPG Collector was killed. In it I found the following sequences;

0042963: 6964: General: RunProgram: Start
0042963: 6964: General: RunProgram: Filename: "C:\Program Files (x86)\Geekzone\EPG Collector\EPGCollector.exe" / Parameters: /ini="C:\Users\htpc\AppData\Local\Geekzone\EPG Collector\EPG CollectorMHEG5+EIT.ini"
0042963: 6964: General: RunProgram: Performing ShellExecute...
0042978: 6964: General: RunProgram: Running process...
0042978: 6964: General: RunProgram: Waiting for completion
.
.
.
0643005: 6964: General: RunProgram: Timeout
0643005: 6964: General: RunProgram: Done waiting
0643005: 6964: General: RunProgram: Finish (600042 ms)


If those numbers on the left are milliseconds, which seems likely now, then 643005 - 42963 = 600042 ms = 10 minutes plus a little for response time.

It does indeed seem that MC times out the EPG Collection process, and kills it at 10 minutes.

The MC log is attached, and I have included in it the EPG Collector log from this run showing the log is simply terminated, and a successful EPG Collector log run manually from within MC showing how an EPG Collector run is supposed to finish.

Can we have a fix for this please?


Logged
What specific version of MC you are running:MC27.0.27 @ Oct 27, 2020 and updating regularly Jim!                        MC Release Notes: https://wiki.jriver.com/index.php/Release_Notes
What OS(s) and Version you are running:     Windows 10 Pro 64bit Version 2004 (OS Build 19041.572).
The JRMark score of the PC with an issue:    JRMark (version 26.0.52 64 bit): 3419
Important relevant info about your environment:     
  Using the HTPC as a MC Server & a Workstation as a MC Client plus some DLNA clients.
  Running JRiver for Android, JRemote2, Gizmo, & MO 4Media on a Sony Xperia XZ Premium Android 9.
  Playing video out to a Sony 65" TV connected via HDMI, playing digital audio out via motherboard sound card, PCIe TV tuner

Sandy B Ridge

  • Citizen of the Universe
  • *****
  • Posts: 884
Re: [EPG] MC Kills my external EPG data collector
« Reply #1 on: September 23, 2014, 04:53:53 am »

I just run EPG Collector from Windows Scheduler and time MC to load the data 20 min later. Works a treat.
Logged

astromo

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 2251
Re: [EPG] MC Kills my external EPG data collector
« Reply #2 on: September 23, 2014, 06:09:22 am »

I just run EPG Collector from Windows Scheduler and time MC to load the data 20 min later. Works a treat.

Now that's upstairs for thinking there Mr Ridge..  ;)

That plan would be assisted by the recent changes around user control of EPG activity.
Logged
MC33, Win10 x64, HD-Plex H5 Gen2 Case, HD-Plex 400W Hi-Fi DC-ATX / AC-DC PSU, Gigabyte Z370 ULTRA Gaming 2.0 MoBo, Intel Core i7 8700 CPU, 4x8GB GSkill DDR4 RAM, Schiit Modi Multibit DAC, Freya Pre, Nelson Pass Aleph J DIY Clone, Ascension Timberwolf 8893BSRTL Speakers, BJC 5T00UP cables, DVB-T Tuner HDHR5-4DT

RoderickGI

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 8186
Re: [EPG] MC Kills my external EPG data collector
« Reply #3 on: September 23, 2014, 07:46:23 am »

I just run EPG Collector from Windows Scheduler and time MC to load the data 20 min later. Works a treat.


That is what I am doing at the moment, but I would like to use MC for the whole process. After all, the function is there, so it should work. I still have to run a Windows Scheduler task to wake the PC before the MC Load Program Guide task as well, as the MC process won't wake the PC, even though it does wake the PC for TV recordings. That is another thing that needs to be fixed, if it isn't just a problem with my PC.

If I wanted to run all sorts of bits and pieces of software and stitch them together, I might have selected XBMC instead of MC. I like that MC is essentially one application, so that it has a greater chance of all working, and should require a lot less maintenance than the other solutions that are made up of bits and pieces.

I'm sure this issue can be fixed, hopefully quickly. Then the boys can get on with improving the whole TV experience. Oh, after one more issue I will document tomorrow. ;D
Logged
What specific version of MC you are running:MC27.0.27 @ Oct 27, 2020 and updating regularly Jim!                        MC Release Notes: https://wiki.jriver.com/index.php/Release_Notes
What OS(s) and Version you are running:     Windows 10 Pro 64bit Version 2004 (OS Build 19041.572).
The JRMark score of the PC with an issue:    JRMark (version 26.0.52 64 bit): 3419
Important relevant info about your environment:     
  Using the HTPC as a MC Server & a Workstation as a MC Client plus some DLNA clients.
  Running JRiver for Android, JRemote2, Gizmo, & MO 4Media on a Sony Xperia XZ Premium Android 9.
  Playing video out to a Sony 65" TV connected via HDMI, playing digital audio out via motherboard sound card, PCIe TV tuner

Yaobing

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 10926
  • Dogs of the world unite!
Re: [EPG] MC Kills my external EPG data collector
« Reply #4 on: September 23, 2014, 10:09:33 am »

This comes down to one question, how long do we wait?

10 minutes seems to be a long time already.  Do you think you can speed up EPG Collector a big?   ;D

I will change it to 20 minutes for the next build.
Logged
Yaobing Deng, JRiver Media Center

RoderickGI

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 8186
Re: [EPG] MC Kills my external EPG data collector
« Reply #5 on: September 23, 2014, 06:50:46 pm »

This comes down to one question, how long do we wait?

10 minutes seems to be a long time already.  Do you think you can speed up EPG Collector a big?   ;D

I will change it to 20 minutes for the next build.

Thanks Yaobing.

I have thought about the issue of how long MC should wait, and there is no simple answer.

For the "internal" MC option, "Download EPG automatically", ten minutes should be plenty of time. Well, depending on server load at the source I guess. I can't use it so I can't test it.  :)
I'm not sure for the mc2xml option, as I haven't used it. It does use an external program, but it is downloading formatted XMLTV files from known sources, and should be quick. So ten minutes should be fine in most cases.
However the XMLTV option uses a fully external and unknown program. It could take any amount of time to complete, and JRiver can't really predict how long that is going to be.


For example, as I mentioned above the basic EPG data collection using EPG Collector (EPGC) takes less than 3 minutes for my 34 channels. But what if I had 300 channels, like some people do? EPGC supports OTA, Cable, and Satellite providers, so 300 channels would be quite possible.

On top of that, if an EPGC user wants to improve their EPG data and they turn on Metadata Lookup, as I did, the time is extended tenfold. EPGC takes a lot longer to run the Metadata Lookup initially, as it builds a database of programs it has previously looked up and only goes to the internet when it doesn't already have the information for a program. Initially, for my 1 week, 34 channel EPG, the Metadata Lookup took 30 minutes or so.

If I have to refresh the EPGC Metadata Databases for any reason, my EPG run would take about 35 minutes for the first few runs. Currently it is stabilising at between 16 and 18 minutes. So 20 minutes is barely enough for me right now, and would not be long enough if anything changes.

But wait, there is more. EPGC also allows the possibility of combining multiple sources of EPG data together as part of its run. I considered using a local high quality EPG data source, but as it doesn't provide data for all my channels, I would still need to look up the remaining channels using the OTA transmission, and then combine the data. That would add still more time.

Finally, it would not be good to have a long global wait time in MC, because if something went wrong it would be a long time before the process is shut down and therefore a long time the user had to wait if they were monitoring the process or testing it as I have been. Not to forget, what would happen if it was still running and a TV recording was due to start?


I think the answer to your question; "How long do we wait?" comes down to; "We don't know."

Therefore, at least for the XMLTV option, I think the best answer is to provide a Wait Parameter, specifying how long MC should wait for the process to complete. I suggest that it could be put into the Load Program Guide dialogue along with the external program settings.


Other issues:
Another question you may ask is; Why is there no process termination time when the EPG process is run manually from within MC, but a process termination time is imposed when it is run on schedule. There must have been some recognition that the EPG collection could take longer than 10 minutes, and when running it manually it should be allowed to run until completion, no matter how long that took.

What other processes are affected by having the Load Program Guide process run a long time? Can the Guide be updated while it is in use? I'm sure you have considered all those sorts of issues in the past Yaobing. I have little idea of the implications.

Why must the EPG data collection wait until no programs are recording? If there are spare tuners, as there are in my case, couldn't it just run anyway? If I run EPGC externally on a Windows Scheduler Task, EPGC just takes the next available tuner anyway.

If a TV recording is due to start, and all tuners will be required by MC and the EPG data collection is still running, should the EPG data collection be terminated and restarted at a time there will be free tuners? If so, how big a gap between recordings is required? Could a Wait Parameter as suggested above be used as the required gap?

Why doesn't MC wake the HTPC from sleep to run the Load Program Guide process? After all, it can wake the HTPC to record a program.


Once again, forgive my long response. But the devil is in the detail, so I need to explain my thought process in asking for changes. Thanks for putting up with me.  :D
Logged
What specific version of MC you are running:MC27.0.27 @ Oct 27, 2020 and updating regularly Jim!                        MC Release Notes: https://wiki.jriver.com/index.php/Release_Notes
What OS(s) and Version you are running:     Windows 10 Pro 64bit Version 2004 (OS Build 19041.572).
The JRMark score of the PC with an issue:    JRMark (version 26.0.52 64 bit): 3419
Important relevant info about your environment:     
  Using the HTPC as a MC Server & a Workstation as a MC Client plus some DLNA clients.
  Running JRiver for Android, JRemote2, Gizmo, & MO 4Media on a Sony Xperia XZ Premium Android 9.
  Playing video out to a Sony 65" TV connected via HDMI, playing digital audio out via motherboard sound card, PCIe TV tuner

astromo

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 2251
Re: [EPG] MC Kills my external EPG data collector
« Reply #6 on: September 23, 2014, 07:14:09 pm »

This comes down to one question, how long do we wait?

10 minutes seems to be a long time already.  Do you think you can speed up EPG Collector a big?   ;D

I will change it to 20 minutes for the next build.
Thanks Yaobing.

I have thought about the issue of how long MC should wait, and there is no simple answer.

...

I think the answer to your question; "How long do we wait?" comes down to; "We don't know."

Therefore, at least for the XMLTV option, I think the best answer is to provide a Wait Parameter, specifying how long MC should wait for the process to complete. I suggest that it could be put into the Load Program Guide dialogue along with the external program settings.

I had the same thought, "make it user configurable".

I know from a dev point of view, more features adds complexity that ends up as more that can go wrong, but if it's included in the Advanced section then at least you've taken steps to avoid idle curiosity.
Logged
MC33, Win10 x64, HD-Plex H5 Gen2 Case, HD-Plex 400W Hi-Fi DC-ATX / AC-DC PSU, Gigabyte Z370 ULTRA Gaming 2.0 MoBo, Intel Core i7 8700 CPU, 4x8GB GSkill DDR4 RAM, Schiit Modi Multibit DAC, Freya Pre, Nelson Pass Aleph J DIY Clone, Ascension Timberwolf 8893BSRTL Speakers, BJC 5T00UP cables, DVB-T Tuner HDHR5-4DT

RoderickGI

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 8186
Re: [EPG] MC Kills my external EPG data collector
« Reply #7 on: October 05, 2014, 10:49:03 pm »

I will change it to 20 minutes for the next build.

Quote from: Media Center 20.0.20 -- Available here
20.0.18 (9/26/2014)
5. Changed: When loading television EPG data in the background, MC will wait longer for external XMLTV EPG programs to complete loading data.

Thanks again Yaobing for making this change. I have now re-implemented running the EPG Collector process from within MC, which is my preferred method. I had been using the Windows Task Scheduler to run it while waiting for this change.

The EPG Collector runs currently take between 18 and 19 1/2 minutes to complete. This is pretty close to the 20 minutes you said you would change it to. If the EPG Collector run takes more than 20 minutes it will again be killed and I will not get EPG updates for that day, and probably not for any following days.

So, did you just change the time limit to 20 minutes in 20.0.18, or did you make it slightly longer, perhaps based on the discussion in this thread?
Logged
What specific version of MC you are running:MC27.0.27 @ Oct 27, 2020 and updating regularly Jim!                        MC Release Notes: https://wiki.jriver.com/index.php/Release_Notes
What OS(s) and Version you are running:     Windows 10 Pro 64bit Version 2004 (OS Build 19041.572).
The JRMark score of the PC with an issue:    JRMark (version 26.0.52 64 bit): 3419
Important relevant info about your environment:     
  Using the HTPC as a MC Server & a Workstation as a MC Client plus some DLNA clients.
  Running JRiver for Android, JRemote2, Gizmo, & MO 4Media on a Sony Xperia XZ Premium Android 9.
  Playing video out to a Sony 65" TV connected via HDMI, playing digital audio out via motherboard sound card, PCIe TV tuner

RoderickGI

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 8186
Re: [EPG] MC Kills my external EPG data collector
« Reply #8 on: October 13, 2014, 05:48:25 pm »

So I see that you did extend the Load Program Guide process time limit to only 20 minutes Yaobing.

I know this because for the last four days my EPG Collector run has been terminated at 20 minutes exactly. I just ran it manually and it completed successfully at just over 23 minutes 16 seconds.

So, for XMLTV specifically, could we please have a User Configurable process timeout?

Thanks.
Logged
What specific version of MC you are running:MC27.0.27 @ Oct 27, 2020 and updating regularly Jim!                        MC Release Notes: https://wiki.jriver.com/index.php/Release_Notes
What OS(s) and Version you are running:     Windows 10 Pro 64bit Version 2004 (OS Build 19041.572).
The JRMark score of the PC with an issue:    JRMark (version 26.0.52 64 bit): 3419
Important relevant info about your environment:     
  Using the HTPC as a MC Server & a Workstation as a MC Client plus some DLNA clients.
  Running JRiver for Android, JRemote2, Gizmo, & MO 4Media on a Sony Xperia XZ Premium Android 9.
  Playing video out to a Sony 65" TV connected via HDMI, playing digital audio out via motherboard sound card, PCIe TV tuner

Yaobing

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 10926
  • Dogs of the world unite!
Re: [EPG] MC Kills my external EPG data collector
« Reply #9 on: October 13, 2014, 09:28:41 pm »


So, for XMLTV specifically, could we please have a User Configurable process timeout?


I think that is what is needed.
Logged
Yaobing Deng, JRiver Media Center

RoderickGI

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 8186
Re: [EPG] MC Kills my external EPG data collector
« Reply #10 on: October 13, 2014, 10:00:48 pm »

Yep. No doubt that is now somewhere on your long list of todo items.  :D

Thanks Yaobing.
Logged
What specific version of MC you are running:MC27.0.27 @ Oct 27, 2020 and updating regularly Jim!                        MC Release Notes: https://wiki.jriver.com/index.php/Release_Notes
What OS(s) and Version you are running:     Windows 10 Pro 64bit Version 2004 (OS Build 19041.572).
The JRMark score of the PC with an issue:    JRMark (version 26.0.52 64 bit): 3419
Important relevant info about your environment:     
  Using the HTPC as a MC Server & a Workstation as a MC Client plus some DLNA clients.
  Running JRiver for Android, JRemote2, Gizmo, & MO 4Media on a Sony Xperia XZ Premium Android 9.
  Playing video out to a Sony 65" TV connected via HDMI, playing digital audio out via motherboard sound card, PCIe TV tuner

muzicman0

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 1239
Re: [EPG] MC Kills my external EPG data collector
« Reply #11 on: October 20, 2014, 01:37:52 pm »

I just noticed mine is timing out now to, using zap2xml.  I was down to just 3 days of guide data before I noticed!  I ran it manually, and it finished, but certainly took longer than 10 minutes.
Logged

RoderickGI

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 8186
Re: [EPG] MC Kills my external EPG data collector
« Reply #12 on: October 20, 2014, 05:01:47 pm »

Yaobing has set the timeout to 20 minutes now, but mine is taking around 26 minutes regularly. When you run it manually the general log tells you how long it took on the fifth last line. The history log breaks the total time down into EPG collection time and metadata lookup time. My EPG collection is still running at about 2.5 minutes. Running manually, even from within MC, never gets a timeout and you can then check the logs for the total time required.

I have moved the EPG Collector run out to Windows Task Scheduler for now, and just run the import process on a schedule inside MC, but I don't like that. After all, if I am going to run stuff outside MC, why not run TV outside completely . . . plus tagging, file conversion, internet lookup, . . . perhaps everything.

It would be nice if there was some reporting of a failed EPG run as well. But EPG Collector can't do it, as it is just killed. MC could report if it didn't get an "Exiting with code = 0" message, or doesn't get an exit code, or knows it killed the process.

I'll leave those improvements to Glynor and Yaobing as they go through an improvement cycle though.
Logged
What specific version of MC you are running:MC27.0.27 @ Oct 27, 2020 and updating regularly Jim!                        MC Release Notes: https://wiki.jriver.com/index.php/Release_Notes
What OS(s) and Version you are running:     Windows 10 Pro 64bit Version 2004 (OS Build 19041.572).
The JRMark score of the PC with an issue:    JRMark (version 26.0.52 64 bit): 3419
Important relevant info about your environment:     
  Using the HTPC as a MC Server & a Workstation as a MC Client plus some DLNA clients.
  Running JRiver for Android, JRemote2, Gizmo, & MO 4Media on a Sony Xperia XZ Premium Android 9.
  Playing video out to a Sony 65" TV connected via HDMI, playing digital audio out via motherboard sound card, PCIe TV tuner
Pages: [1]   Go Up