Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[hoyt]

I have a python script that manages my imports to MC.  I've noticed that MCWS seems "slow" when I look at the log files and wondered if there was a better way than what I was doing to keep a connection open to my server.  I have this in my python script:

Code: [Select]

class mcwsCall:
def __init__(self, call, params, respType="XML"):
self.call = call
self.params = params
self.respType = respType
logging.info("in mcwsCall: {0}".format(self.call))
logging.debug("-params: {0}".format(self.params))
header = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
mcws = "{0}://{1}:{2}/MCWS/v1/".format(cfg['https'],cfg['host'],cfg['port'])
urlparam = ""
if(len(self.params) > 0):
for k in self.params.keys():
urlparam = urlparam + urllib.parse.quote(k,safe='[]();/?:@=+$,') + '=' + urllib.parse.quote(str(self.params[k]),safe='[]();/?:@=+$,') + '&'
urlparam = urlparam[:-1]
mcws = mcws + self.call + "?{0}".format(urlparam)
logging.info("-fullCall: {0}".format(mcws))
response = requests.get(mcws,headers=header,auth = HTTPBasicAuth(cfg["username"],cfg["password"]))
if self.respType == "JSON":
response = response.json()
elif self.respType == "XML":
root = ElementTree.fromstring(response.content)
response = {root.tag:root.attrib}
response["items"] = []
for child in root:
childDict = {child.attrib["Name"]:child.text}
response["items"].append(childDict)
self.response = response
logging.debug("-response: {0}".format(response))

And throughout my code, when I need to call into MCWS I do something like this:

Code: [Select]

params={"Block":"1","path":file}
call = mcwsCall("Library/Import",params,"XML")
if call.response["Response"]["Status"] != "OK":
dieGraceful("Failed to import file! {0}.".format(file))
if (((settings["analyzeAudioForAudio"] == 1 and mimeType == 'audio')
or (settings["analyzeAudioForVideo"] == 1 and mimeType == 'video'))
and newFileInd == 1):
analyzeAudio = 1
if newFileInd == 1:
params = {"Action":"JSON","Fields":"Key","Query":"[Filename]={0}".format(cleanedFileName)}
call = mcwsCall("Files/Search",params,"JSON")

What I'm noticing is that my log files look like this (I cut a lot of lines out to make it cleaner):

Code: [Select]

2022-02-25 14:17:27,047 INFO: in mcwsCall: Files/Search
2022-02-25 14:17:27,047 DEBUG: -params: {'Action': 'JSON', 'Fields': 'Key', 'Query': '[Filename]=Y:\\verified\\2022\\d+t2022-02-20.akgc414xlii.flac24\\d+t2022-02-20.akgc414xlii.t08.flac'}
2022-02-25 14:17:27,047 INFO: -fullCall: http://localhost:52199/MCWS/v1/Files/Search?Action=JSON&Fields=Key&Query=[Filename]=Y:%5Cverified%5C2022%5Cd+t2022-02-20.akgc414xlii.flac24%5Cd+t2022-02-20.akgc414xlii.t08.flac
2022-02-25 14:17:27,049 DEBUG: Starting new HTTP connection (1): localhost:52199
2022-02-25 14:17:29,121 DEBUG: http://localhost:52199 "GET /MCWS/v1/Files/Search?Action=JSON&Fields=Key&Query=%5BFilename%5D=Y:%5Cverified%5C2022%5Cd+t2022-02-20.akgc414xlii.flac24%5Cd+t2022-02-20.akgc414xlii.t08.flac HTTP/1.1" 200 2
2022-02-25 14:17:29,122 DEBUG: -response: []
2022-02-25 14:17:29,122 INFO: in mcwsCall: Library/Import
2022-02-25 14:17:29,122 DEBUG: -params: {'Block': '1', 'path': 'Y:\\verified\\2022\\d+t2022-02-20.akgc414xlii.flac24\\d+t2022-02-20.akgc414xlii.t08.flac'}
2022-02-25 14:17:29,122 INFO: -fullCall: http://localhost:52199/MCWS/v1/Library/Import?Block=1&path=Y:%5Cverified%5C2022%5Cd+t2022-02-20.akgc414xlii.flac24%5Cd+t2022-02-20.akgc414xlii.t08.flac
2022-02-25 14:17:29,123 DEBUG: Starting new HTTP connection (1): localhost:52199
2022-02-25 14:17:33,928 DEBUG: http://localhost:52199 "GET /MCWS/v1/Library/Import?Block=1&path=Y:%5Cverified%5C2022%5Cd+t2022-02-20.akgc414xlii.flac24%5Cd+t2022-02-20.akgc414xlii.t08.flac HTTP/1.1" 200 83
2022-02-25 14:17:33,929 DEBUG: -response: {'Response': {'Status': 'OK'}, 'items': []}
2022-02-25 14:17:33,929 INFO: in mcwsCall: Files/Search
2022-02-25 14:17:33,929 DEBUG: -params: {'Action': 'JSON', 'Fields': 'Key', 'Query': '[Filename]=Y:\\verified\\2022\\d+t2022-02-20.akgc414xlii.flac24\\d+t2022-02-20.akgc414xlii.t08.flac'}
2022-02-25 14:17:33,929 INFO: -fullCall: http://localhost:52199/MCWS/v1/Files/Search?Action=JSON&Fields=Key&Query=[Filename]=Y:%5Cverified%5C2022%5Cd+t2022-02-20.akgc414xlii.flac24%5Cd+t2022-02-20.akgc414xlii.t08.flac
2022-02-25 14:17:33,931 DEBUG: Starting new HTTP connection (1): localhost:52199
2022-02-25 14:17:36,012 DEBUG: http://localhost:52199 "GET /MCWS/v1/Files/Search?Action=JSON&Fields=Key&Query=%5BFilename%5D=Y:%5Cverified%5C2022%5Cd+t2022-02-20.akgc414xlii.flac24%5Cd+t2022-02-20.akgc414xlii.t08.flac HTTP/1.1" 200 17
2022-02-25 14:17:36,012 DEBUG: -response: [{'Key': 7929017}]
2022-02-25 14:17:36,012 INFO: in mcwsCall: File/SetInfo
2022-02-25 14:17:36,012 DEBUG: -params: {'File': 7929017, 'List': 'CSV', 'Field': 'Season,Type - Grouping,Series,Source', 'Value': '2022-02-20,flac24,2022,akgc414xlii'}
2022-02-25 14:17:36,012 INFO: -fullCall: http://localhost:52199/MCWS/v1/File/SetInfo?File=7929017&List=CSV&Field=Season,Type%20-%20Grouping,Series,Source&Value=2022-02-20,flac24,2022,akgc414xlii
2022-02-25 14:17:36,014 DEBUG: Starting new HTTP connection (1): localhost:52199
2022-02-25 14:17:38,068 DEBUG: http://localhost:52199 "GET /MCWS/v1/File/SetInfo?File=7929017&List=CSV&Field=Season,Type%20-%20Grouping,Series,Source&Value=2022-02-20,flac24,2022,akgc414xlii HTTP/1.1" 200 83
2022-02-25 14:17:38,069 DEBUG: -response: {'Response': {'Status': 'OK'}, 'items': []}
2022-02-25 14:17:38,071 DEBUG: URL being requested: PATCH https://www.googleapis.com/drive/v3/files/redactedFileId?supportsAllDrives=true&alt=json
2022-02-25 14:17:38,619 INFO: fullPath: Y:\verified\2022\d+t2022-02-20.akgc414xlii.flac24\d+t2022-02-20.akgc414xlii.t08.flac

Each time I call into MCWS, you can see a new "Starting new HTTP connection" item and it takes a few seconds.  Whereas if I do that same Files/Search in a local browser, it's milliseconds.  When I look in MC, the response time is always milliseconds, whether or not it's within my script or a browser.  Is there a better way that I should be approaching this?  Thanks!

[Wheaten]

Every time you call your script you need to setup the connection, post/get information.
you can use "socket" instead of building a rest connection, which will go faster.

Code: [Select]

import requests
import xml.etree.ElementTree as ET

import socket

def get_ip():
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    try:
        # doesn't even have to be reachable
        s.connect(('10.255.255.255', 1))
        IP = s.getsockname()[0]
    except Exception:
        IP = '127.0.0.1'
    finally:
        s.close()
    return IP

sURL = 'http://' + get_ip() + ':52199/MCWS/v1/Playback/Info?Zone=-1'
sUsername = 'username'
sPassword = 'password'

#avoid looping through the full response just get the needed attribute
def readattrval(roota,branch):
for child in roota.findall(branch):
if debug == 1:
print( child.attrib, child.text)
return child.text

def geturl(url):
#If you need credetials to access the media network use this line
r = requests.get(url, auth=(sUsername, sPassword))
#If you don't need credentials, use this line
#r = requests.get(url)
root = ET.fromstring(r.content)
        global Artist
Artist = readattrval(root,"./Item[@Name='Artist']")

[hoyt]

Every time you call your script you need to setup the connection, post/get information.
you can use "socket" instead of building a rest connection, which will go faster.

Code: [Select]

import requests
import xml.etree.ElementTree as ET

import socket

def get_ip():
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    try:
        # doesn't even have to be reachable
        s.connect(('10.255.255.255', 1))
        IP = s.getsockname()[0]
    except Exception:
        IP = '127.0.0.1'
    finally:
        s.close()
    return IP

sURL = 'http://' + get_ip() + ':52199/MCWS/v1/Playback/Info?Zone=-1'
sUsername = 'username'
sPassword = 'password'

#avoid looping through the full response just get the needed attribute
def readattrval(roota,branch):
for child in roota.findall(branch):
if debug == 1:
print( child.attrib, child.text)
return child.text

def geturl(url):
#If you need credetials to access the media network use this line
r = requests.get(url, auth=(sUsername, sPassword))
#If you don't need credentials, use this line
#r = requests.get(url)
root = ET.fromstring(r.content)
        global Artist
Artist = readattrval(root,"./Item[@Name='Artist']")

Thanks for the quick response!  I'll take a look at this in more detail later.

[Wheaten]

You can also take a look here:
https://yabb.jriver.com/interact/index.php/topic,123397.0.html

[hoyt]

I took some time to walk through this today and perhaps am more confused that before.  In the example snippet that you provided, is that using the socket that it sets up?  I read through a bunch of socket tutorials on google and found other actions like socket.send(), etc.  I built a quick example and I'm not sure I see a difference in what's happening in the log:

Code: [Select]

import socket
import requests
import sys
import logging
import json
from requests.auth import HTTPBasicAuth

logging.basicConfig(filename='socket_test.log', filemode='a', format='%(asctime)s %(levelname)s: %(message)s', level='DEBUG')
logging.debug('run')

with open('watch_folder_config.json', 'r') as cfgfile:
cfg = json.load(cfgfile)
def get_ip():
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
try:
s.connect(('127.0.0.1', 1))
IP = s.getsockname()[0]
print("tried")
except Exception:
print("failed")
IP = cfg['host']
finally:
s.close()
return IP
def get_mc():
print("in get_mc()")
path = '/Alive'
logging.debug('socket?')
r = requests.get('http://' + get_ip() + ':' + cfg['port'] + '/MCWS/v1' + path, auth=(cfg['username'],cfg['password']))
logging.debug('requests.get')
mcws =  "{0}://{1}:{2}/MCWS/v1".format(cfg['https'],cfg['host'],cfg['port'])
response = requests.get(mcws + path,auth = HTTPBasicAuth(cfg['username'],cfg['password']))
def main():
for x in range(10):
get_mc()
if __name__ == '__main__':
main()
sys.exit(0)

My logging looks like this:

Code: [Select]

2022-03-01 14:22:07,933 DEBUG: run
2022-03-01 14:22:07,933 DEBUG: socket?
2022-03-01 14:22:07,935 DEBUG: Starting new HTTP connection (1): 127.0.0.1:52199
2022-03-01 14:22:07,937 DEBUG: http://127.0.0.1:52199 "GET /MCWS/v1/Alive HTTP/1.1" 200 466
2022-03-01 14:22:07,938 DEBUG: requests.get
2022-03-01 14:22:07,939 DEBUG: Starting new HTTP connection (1): 127.0.0.1:52199
2022-03-01 14:22:07,941 DEBUG: http://127.0.0.1:52199 "GET /MCWS/v1/Alive HTTP/1.1" 200 466
2022-03-01 14:22:07,941 DEBUG: socket?
2022-03-01 14:22:07,942 DEBUG: Starting new HTTP connection (1): 127.0.0.1:52199
2022-03-01 14:22:07,944 DEBUG: http://127.0.0.1:52199 "GET /MCWS/v1/Alive HTTP/1.1" 200 466
2022-03-01 14:22:07,945 DEBUG: requests.get
2022-03-01 14:22:07,946 DEBUG: Starting new HTTP connection (1): 127.0.0.1:52199
2022-03-01 14:22:07,948 DEBUG: http://127.0.0.1:52199 "GET /MCWS/v1/Alive HTTP/1.1" 200 466
2022-03-01 14:22:07,948 DEBUG: socket?
2022-03-01 14:22:07,949 DEBUG: Starting new HTTP connection (1): 127.0.0.1:52199
2022-03-01 14:22:07,951 DEBUG: http://127.0.0.1:52199 "GET /MCWS/v1/Alive HTTP/1.1" 200 466
2022-03-01 14:22:07,951 DEBUG: requests.get
2022-03-01 14:22:07,952 DEBUG: Starting new HTTP connection (1): 127.0.0.1:52199
2022-03-01 14:22:07,954 DEBUG: http://127.0.0.1:52199 "GET /MCWS/v1/Alive HTTP/1.1" 200 466
2022-03-01 14:22:07,955 DEBUG: socket?
2022-03-01 14:22:07,956 DEBUG: Starting new HTTP connection (1): 127.0.0.1:52199
2022-03-01 14:22:07,958 DEBUG: http://127.0.0.1:52199 "GET /MCWS/v1/Alive HTTP/1.1" 200 466
2022-03-01 14:22:07,959 DEBUG: requests.get
2022-03-01 14:22:07,960 DEBUG: Starting new HTTP connection (1): 127.0.0.1:52199
2022-03-01 14:22:07,962 DEBUG: http://127.0.0.1:52199 "GET /MCWS/v1/Alive HTTP/1.1" 200 466
2022-03-01 14:22:07,962 DEBUG: socket?
2022-03-01 14:22:07,964 DEBUG: Starting new HTTP connection (1): 127.0.0.1:52199
2022-03-01 14:22:07,966 DEBUG: http://127.0.0.1:52199 "GET /MCWS/v1/Alive HTTP/1.1" 200 466
2022-03-01 14:22:07,966 DEBUG: requests.get
2022-03-01 14:22:07,967 DEBUG: Starting new HTTP connection (1): 127.0.0.1:52199
2022-03-01 14:22:07,969 DEBUG: http://127.0.0.1:52199 "GET /MCWS/v1/Alive HTTP/1.1" 200 466
2022-03-01 14:22:07,970 DEBUG: socket?
2022-03-01 14:22:07,971 DEBUG: Starting new HTTP connection (1): 127.0.0.1:52199
2022-03-01 14:22:07,973 DEBUG: http://127.0.0.1:52199 "GET /MCWS/v1/Alive HTTP/1.1" 200 466
2022-03-01 14:22:07,973 DEBUG: requests.get
2022-03-01 14:22:07,974 DEBUG: Starting new HTTP connection (1): 127.0.0.1:52199
2022-03-01 14:22:07,976 DEBUG: http://127.0.0.1:52199 "GET /MCWS/v1/Alive HTTP/1.1" 200 466
2022-03-01 14:22:07,976 DEBUG: socket?
2022-03-01 14:22:07,978 DEBUG: Starting new HTTP connection (1): 127.0.0.1:52199
2022-03-01 14:22:07,980 DEBUG: http://127.0.0.1:52199 "GET /MCWS/v1/Alive HTTP/1.1" 200 466
2022-03-01 14:22:07,980 DEBUG: requests.get
2022-03-01 14:22:07,981 DEBUG: Starting new HTTP connection (1): 127.0.0.1:52199
2022-03-01 14:22:07,983 DEBUG: http://127.0.0.1:52199 "GET /MCWS/v1/Alive HTTP/1.1" 200 466
2022-03-01 14:22:07,983 DEBUG: socket?
2022-03-01 14:22:07,985 DEBUG: Starting new HTTP connection (1): 127.0.0.1:52199
2022-03-01 14:22:07,988 DEBUG: http://127.0.0.1:52199 "GET /MCWS/v1/Alive HTTP/1.1" 200 466
2022-03-01 14:22:07,989 DEBUG: requests.get
2022-03-01 14:22:07,990 DEBUG: Starting new HTTP connection (1): 127.0.0.1:52199
2022-03-01 14:22:07,991 DEBUG: http://127.0.0.1:52199 "GET /MCWS/v1/Alive HTTP/1.1" 200 466
2022-03-01 14:22:07,991 DEBUG: socket?
2022-03-01 14:22:07,993 DEBUG: Starting new HTTP connection (1): 127.0.0.1:52199
2022-03-01 14:22:07,994 DEBUG: http://127.0.0.1:52199 "GET /MCWS/v1/Alive HTTP/1.1" 200 466
2022-03-01 14:22:07,995 DEBUG: requests.get
2022-03-01 14:22:07,996 DEBUG: Starting new HTTP connection (1): 127.0.0.1:52199
2022-03-01 14:22:07,998 DEBUG: http://127.0.0.1:52199 "GET /MCWS/v1/Alive HTTP/1.1" 200 466
2022-03-01 14:22:07,998 DEBUG: socket?
2022-03-01 14:22:08,000 DEBUG: Starting new HTTP connection (1): 127.0.0.1:52199
2022-03-01 14:22:08,002 DEBUG: http://127.0.0.1:52199 "GET /MCWS/v1/Alive HTTP/1.1" 200 466
2022-03-01 14:22:08,002 DEBUG: requests.get
2022-03-01 14:22:08,003 DEBUG: Starting new HTTP connection (1): 127.0.0.1:52199
2022-03-01 14:22:08,005 DEBUG: http://127.0.0.1:52199 "GET /MCWS/v1/Alive HTTP/1.1" 200 466
How is what you posted any different besides the initial setup of the socket?  In the logs, it certainly looks the same to me ("Starting new HTTP connection, etc").  Am I thinking about this wrong?  I expected to see something different between the socket lines and the requests.get lines (which is what I had originally).  I noticed a drastic difference from changing my config file from "localhost" to "127.0.0.1", which surprised me.  Thanks!

[Wheaten]

in your opening post, you mentioned that opening the connection took several seconds, now it takes a few ms. I expect this was your issue? If not then I didn't understand your problem.
You can leave the connection open, but then you need to create a while True: loop with a sleep(0.3) to reduce CPU load, to handle your code and place the connection outside the loop.
In this case you only connects once, instead of doing this in every call.

Code: [Select]

make the connection

while True:
   Your cosing
   sleep(0.3)

[mattkhan]

requests supports keep alive using its session

https://2.python-requests.org/en/master/user/advanced/#keep-alive

Decouple the call from the collection so you can cache a session then reuse it for multiple calls

[Wheaten]

for sure a cleaner option.

[hoyt]

in your opening post, you mentioned that opening the connection took several seconds, now it takes a few ms.

Yeah, but when I changed my original code to use 127.0.0.1 instead of localhost, it changed to a few ms too.  That's why I was trying to compare what you had suggested to figure out if there was something I failed to work into the script.

requests supports keep alive using its session

https://2.python-requests.org/en/master/user/advanced/#keep-alive

Decouple the call from the collection so you can cache a session then reuse it for multiple calls

Thanks!  Will read up on this over the next few days and try to incorporate it.

I expect this was your issue? If not then I didn't understand your problem.

My main concern is that I move dozens of folders into my MC media at a time.  I keep track of live recordings and during the summer touring session, I'll move a week's worth of recordings at a time, which could be 5-700 tracks.  So this script kicks off and finds these few hundred files and if each file is taking 10 seconds, this drags and drags. The /Import/ call seems like it's always going to take a few seconds, and not sure I can get around that (because I have the block parameter set so I can get the FileKey back).

[hoyt]

Ah, this is what I was expecting to see when the socket was mentioned:

Code: [Select]

def get_mc(session):
print("in get_mc()")
path = '/Alive'
logging.debug('requests.get')
mcws =  "{0}://{1}:{2}/MCWS/v1".format(cfg['https'],cfg['host'],cfg['port'])
response = session.get(mcws + path,auth = HTTPBasicAuth(cfg['username'],cfg['password']))
def main():
session = requests.Session()
for x in range(10):
get_mc(session)


Code: [Select]

2022-03-01 15:42:16,027 DEBUG: run
2022-03-01 15:42:16,028 DEBUG: requests.get
2022-03-01 15:42:16,029 DEBUG: Starting new HTTP connection (1): 127.0.0.1:52199
2022-03-01 15:42:16,033 DEBUG: http://127.0.0.1:52199 "GET /MCWS/v1/Alive HTTP/1.1" 200 466
2022-03-01 15:42:16,033 DEBUG: requests.get
2022-03-01 15:42:16,035 DEBUG: http://127.0.0.1:52199 "GET /MCWS/v1/Alive HTTP/1.1" 200 466
2022-03-01 15:42:16,036 DEBUG: requests.get
2022-03-01 15:42:16,038 DEBUG: http://127.0.0.1:52199 "GET /MCWS/v1/Alive HTTP/1.1" 200 466
2022-03-01 15:42:16,038 DEBUG: requests.get
2022-03-01 15:42:16,040 DEBUG: http://127.0.0.1:52199 "GET /MCWS/v1/Alive HTTP/1.1" 200 466
2022-03-01 15:42:16,041 DEBUG: requests.get
2022-03-01 15:42:16,043 DEBUG: http://127.0.0.1:52199 "GET /MCWS/v1/Alive HTTP/1.1" 200 466
2022-03-01 15:42:16,043 DEBUG: requests.get
2022-03-01 15:42:16,045 DEBUG: http://127.0.0.1:52199 "GET /MCWS/v1/Alive HTTP/1.1" 200 466
2022-03-01 15:42:16,046 DEBUG: requests.get
2022-03-01 15:42:16,048 DEBUG: http://127.0.0.1:52199 "GET /MCWS/v1/Alive HTTP/1.1" 200 466
2022-03-01 15:42:16,049 DEBUG: requests.get
2022-03-01 15:42:16,051 DEBUG: http://127.0.0.1:52199 "GET /MCWS/v1/Alive HTTP/1.1" 200 466
2022-03-01 15:42:16,052 DEBUG: requests.get
2022-03-01 15:42:16,054 DEBUG: http://127.0.0.1:52199 "GET /MCWS/v1/Alive HTTP/1.1" 200 466
2022-03-01 15:42:16,055 DEBUG: requests.get
2022-03-01 15:42:16,057 DEBUG: http://127.0.0.1:52199 "GET /MCWS/v1/Alive HTTP/1.1" 200 466

Also - I updated the titled of the thread because I thought it looked like I was complaining that MCWS was slow, but I really meant that it was slow within my scripting.