Topic: mc2xml v1.5 error - "Did not receive User-Agent in header" [SOLVED] (Read 12149 times)

CountryBumkin · « **on:** November 30, 2015, 07:46:15 am »

I found today that mc2xml has not run since before the holiday. I am running the standalone version 1.5 using the Schedules Direct JSON. I don't know if this also affects mc2xml v1.4 (someone may want to check).

When I manually run the program, it starts normally in the Win10 Command Prompt window then after a few lines, lists the error message "Did not receive User-Agent header" and ends. I deleted the mc2xml.dat file and reran the program entering all my user info again, but it still comes up with the same error.

I'm still searching for an answer on this - but wanted to post here now in case someone has this issue and/or knows what is going on.

I checked the mc2xml site and Schedules Direct site, but there is nothing listed about this error.
However, there is a post on the SD forum about a Win10 authentication issue due to a prior Win10 update which is supposed to fix with the November update (Win10 Home version 1511, 10586) - so I'm running that update now.

JimH · « **Reply #1 on:** November 30, 2015, 09:47:25 am »

Firewall?

CountryBumkin · « **Reply #2 on:** November 30, 2015, 10:16:41 am »

I'm not sure yet. But it worked fine for several weeks without my interaction until a few days ago. I have not changed anything - but not sure if Win10 did something behind my back. Could be that something with Schedules Direct changed/broke.

I'm sure it is not an MC issue - I'm just posting here to get some computer expert feed back.

CountryBumkin · « **Reply #3 on:** December 01, 2015, 05:05:52 am »

Issue solved. I suppose a Windows update caused the problem but I'm not sure.
Anyways, I had to go to Windows Control Panel>System>Advanced System Settings>Environment Variables and add "Useragent = my e-mail address".

RoderickGI · « **Reply #4 on:** December 01, 2015, 02:43:31 pm »

The cause could have been security software removing your email address from the environment variable.

I know at least one of the Symantec identity protect applications could be very aggressive in removing personal data such as email addresses from certain areas in my computer. When I told it to protect my email address it caused all sorts of grief. I had to tone it down or turn it off, I can't remember which. Anyway, that functionality has since been removed from Norton 360, which I use now.

Of course, having your email address in an environment variable may expose it quite a bit when you are online. I'm not sure if web sites can see environment variables, but if they look, it is possible. Perhaps check that out.

Also, sometimes if some information is not available via a Registry setting, it can be placed into an environment variable as an alternative. Perhaps check with the author of mc2xml whether this is the case, and how to avoid using an environment variable.

CountryBumkin · « **Reply #5 on:** December 01, 2015, 02:50:54 pm »

Thanks for the heads-up.
Actually I didn't know my e-mail address was ever in Environment Variables. I didn't put it there myself - must have been done through the mc2xml or Schedules Direct JSON install.
What is the risk of having my e-mail address exposed? Just more spam - or something more dangerous?

glynor · « **Reply #6 on:** December 01, 2015, 05:27:06 pm »

Quote from: CountryBumkin on December 01, 2015, 02:50:54 pm

What is the risk of having my e-mail address exposed? Just more spam - or something more dangerous?

Does it work if you put yeah@right.com in there? Useragent wouldn't typically even be an email address, and why would it pull from the environment variables? I'm kinda confused by this solution.

For the record, my Windows 8.1 install has never seen any AV or security suites other than Windows Defender (nothing that would do anything like this, anyway) and I have no such environment variable on my system. I can check Windows 10 later.

RoderickGI · « **Reply #7 on:** December 01, 2015, 05:32:19 pm »

I am probably being a little paranoid, and something would have to actually get onto your PC to read the data. But maybe a web site you visit could read environment variables as well. I don't know. Haven't looked into it.

If something can read the variable, then yes, potentially more SPAM, potentially the first step in identity theft. Probably no worse than other methods that spammers and scammers get hold of your email address though.

I would be asking the mc2xml developer what the message means, and why it came up now.

But in the mean time, it looks like you can use any text for the UserAgent value. It doesn't have to be your email address. So change it to something else.

From the mc2xml site.

Quote

If you want to use a user-agent string you can set the USERAGENT environment variable (set USERAGENT=email or any string) before running mc2xml (requires latest version).

glynor · « **Reply #8 on:** December 01, 2015, 05:36:51 pm »

Quote from: RoderickGI on December 01, 2015, 05:32:19 pm

I am probably being a little paranoid, and something would have to actually get onto your PC to read the data. But maybe a web site you visit could read environment variables as well. I don't know. Haven't looked into it.

No. The error message you are getting is about a HTTP Request Header. This is what is sent to EVERY WEBSITE YOU CONNECT TO with that particular browser. I don't know what useragent mc2xml uses, but I'd guess the IE one (or it has one hardcoded, but I guess not or you wouldn't be getting this error).

I have no idea how an environment variable "fixed" it, unless you have your User-Agent registry keys pointing at an environment variable somehow. More:
http://stackoverflow.com/questions/15069533/http-request-header-useragent-variable
https://msdn.microsoft.com/en-us/library/ms537503%28v=vs.85%29.aspx#UARegistry

In any case, this information absolutely is being transmitted (probably via HTTP in the clear), at least via mc2xml, on every request, otherwise it wouldn't have solved the error.

glynor · « **Reply #9 on:** December 01, 2015, 05:45:03 pm »

I found more. That environment variable is specific to mc2xml. It isn't clear when it is needed, but it is mentioned in the docs for mc2xml here under Schedules Direct: http://mc2xml.awardspace.info/

Quote

If you want to use a user-agent string you can set the USERAGENT environment variable (set USERAGENT=email or any string) before running mc2xml (requires latest version)

So, this is something specific to how mc2xml works. They do say, set it to your email or any string. I'd probably put in "mah mc2xml" or the user-agent from Firefox or something and see if it works.

Basically all environment variables are sandboxed in Javascript and Java, at least. Not doing so would be a massive security hole. I know in some cases Javascript can read %TEMP% indirectly, but it shouldn't be able to just dump them all out.

I think it is safe, relatively, except that mc2xml will be transmitting it to wherever it connects.

glynor · « **Reply #10 on:** December 01, 2015, 06:03:33 pm »

And, of course, all other software executing locally on your PC can read out the value.

Why would the mc2xml authors use an environment variable for this? Why wouldn't they just pick a user agent (or transmit the IE one) and hardcode it? Or read it from an app specific XML or INI file or something?

Whole thing is weird. User Agent is to identify the browser to the remote webserver, not the user.

muzicman0 · « **Reply #11 on:** December 01, 2015, 06:09:56 pm »

Just FYI...same thing happened to me...thanks for the fix!

I entered my initials as the string, and it worked fine.

glynor · « **Reply #12 on:** December 01, 2015, 06:14:51 pm »

I'm sure, then, it'd work with any string. I'd just use mc2xml. That's reporting to them what User-Agent should be reporting, anyway.

RoderickGI · « **Reply #13 on:** December 01, 2015, 06:20:40 pm »

Maybe the developer is trying to bypass some download restriction per day or similar by allowing the user to set the variable. Hit the limit, change the variable, resets the count, away you go again. Except I would have thought limits would be set by User ID, which still needs to be sent.

It doesn't seem like it is supposed to be compulsory to set the variable, but something changed to make it so.

As above, I would ask the developer. He responds . . . eventually.

glynor · « **Reply #14 on:** December 01, 2015, 06:58:47 pm »

Quote from: RoderickGI on December 01, 2015, 06:20:40 pm

Maybe the developer is trying to bypass some download restriction per day or similar by allowing the user to set the variable. Hit the limit, change the variable, resets the count, away you go again. Except I would have thought limits would be set by User ID, which still needs to be sent.

Yeah. That doesn't make any sense. The User-Agent string is supposed to uniquely identify the browser (and optionally more details about the platform), for rendering compatibility purposes (so that a server can send a page optimized for the browser). For example, this is mine right now as I post this:

Code: [Select]

Mozilla/5.0 (Windows NT 6.3; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0
I'm sure the Schedules Direct server just updated to some different code on their server (probably an analytics package) and now their webserver refuses connections from clients which don't provide a user-agent string.

In fact, by using your initials or email address or whatever, you're probably just mucking up their analytics.

muzicman0 · « **Reply #15 on:** December 01, 2015, 07:16:53 pm »

per this page:

https://github.com/SchedulesDirect/JSON-Service/wiki/API-20140530

I believe it should be the actual grabber that is used...but I believe it should be set in the application (mc2xml in this case). It's used so that SD can help a developer troubleshoot. Since I have no idea what to put there, I will leave it as is. I suppose I could put mc2xml, but I don't really know the correct format that they want the string to be in...

glynor · « **Reply #16 on:** December 01, 2015, 08:28:29 pm »

Yes. I just read that too. The grabber is the "browser" in this instance.

Put mc2xml in there. The developer of mc2xml should be doing this themselves, according to the SchedulesDirect API, but they fail at their job I guess. If you want, put mc2xml and your version in there (but then you have to update it manually every time you get a new mc2xml version).

And, if you're active on their forum or whatever, tell the dev to fix it and point them at the API documentation.

CountryBumkin · « **Reply #17 on:** December 02, 2015, 11:14:55 am »

Thanks for your response on this.
I went back and changed USERAGENT = "my email" to USERAGENT = mc2xml, and the program still runs fine. So my e-mail is no longer exposed to the world.

greynolds · « **Reply #18 on:** May 02, 2016, 06:31:54 am »

Adding the USERAGENT environment variable isn't working for me on Windows 10. I've tried adding it as a system variable and as a user variable. I haven't tried rebooting after adding the variable, but that shouldn't be necessary as long as I start a new MC2XML instance after adding the variable.

CountryBumkin · « **Reply #19 on:** May 02, 2016, 06:54:21 am »

I'm not sure about whether you need to reboot or not, but the easiest thing would be to reboot and see if that takes care of it.
It's not Win10 problem, since I am on Win10 too, and the UserAgent fix worked for me.

In my Win10 "Environment Variables", I have
USERAGENT mc2xml
USERNAME system

greynolds · « **Reply #20 on:** May 02, 2016, 07:20:35 am »

Thanks! It looks like a reboot did help in this case, but I had to add them as System variables, not User variables in order to get it to work.

Next step is to update my MC system to use this data instead of PercData.

imeric · « **Reply #21 on:** May 03, 2016, 12:43:16 pm »

As an FYI I'm running W10 and I never had to specify or change my environment variables or a USERAGENT with my "external within MC..." mc2xml setup. It's been working fine with 2 JSON EPG profiles for weeks now.

One thing I do however is specify a full command line for both profiles in MC.

For example:

Code: [Select]

-d 504 -D M:\mc2xml_SHAW\mc2xml.dat -C M:\mc2xml_SHAW\mc2xml.chl -o M:\mc2xml_SHAW\xmltv.xml
From past experiences if I don't do this MC seems to get confused either between the 2 profiles and or the internal mc2xml and it never worked properly without it...

I currently use mc2xml v1.6 (make sure you go download the latest as it was changed recently for issues with JSON and it's not publicised on the site)
I'm running 21.0.76

greynolds · « **Reply #22 on:** May 03, 2016, 01:11:57 pm »

Quote from: imeric on May 03, 2016, 12:43:16 pm

As an FYI I'm running W10 and I never had to specify or change my environment variables or a USERAGENT with my "external within MC..." mc2xml setup. It's been working fine with 2 JSON EPG profiles for weeks now.

One thing I do however is specify a full command line for both profiles in MC.

For example:
Code: [Select]
-d 504 -D M:\mc2xml_SHAW\mc2xml.dat -C M:\mc2xml_SHAW\mc2xml.chl -o M:\mc2xml_SHAW\xmltv.xml
From past experiences if I don't do this MC seems to get confused either between the 2 profiles and or the internal mc2xml and it never worked properly without it...

Interesting. The 2 PC's I've tried MC2XML on seemed to need those environment variables set.

Quote from: imeric on May 03, 2016, 12:43:16 pm

I currently use mc2xml v1.6 (make sure you go download the latest as it was changed recently for issues with JSON and it's not publicised on the site)
I'm running 21.0.76

It obviously doesn't matter for the SD JSON option, but did you make a donation for the new WMC Rovi option? If so, did you need to ask for a special version of 1.6 that includes that feature or does it have some way of knowing who donated?

imeric · « **Reply #23 on:** May 03, 2016, 02:10:35 pm »

As a donator he sent me another .exe But I'm not using it.

imeric · « **Reply #24 on:** May 04, 2016, 02:32:30 pm »

There is a JSON grabber one can use to tweak your JSON profile (s). I did use this at some point maybe this could help...

INTERACT FORUM

Author Topic: mc2xml v1.5 error - "Did not receive User-Agent in header" [SOLVED] (Read 12149 times)