INTERACT FORUM

Please login or register.

Login with username, password and session length
Advanced search  
Pages: [1]   Go Down

Author Topic: Publish / Subscribe for changes?  (Read 1916 times)

masterjoe

  • World Citizen
  • ***
  • Posts: 235
Publish / Subscribe for changes?
« on: August 26, 2024, 06:23:24 am »

Hi there,

currently for automation (using MCWS etc) it is required to POLL for changes. This is not very efficient and changes are delayed when you do not want to hammer too hard.

It would be awesome to have a Publish / Subscribe mechanism with an OPEN TCP conection that pushes changes when they happen to subscribed clients (and topics).

Eventually something like MQTT would be nice but a proprietary mechanism in MC would also be okay.

Subscribable topics would include:
- file selection changes
- file playback changes (file, position etc.)
- new files added (import)
etc.

Would be super cool  :)
Logged

JimH

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 72443
  • Where did I put my teeth?
Re: Publish / Subscribe for changes?
« Reply #1 on: August 26, 2024, 06:47:24 am »

It would be awesome to have a Publish / Subscribe mechanism with an OPEN TCP conection that pushes changes when they happen to subscribed clients (and topics).

Eventually something like MQTT would be nice but a proprietary mechanism in MC would also be okay.
Thanks.  I had to look up MQTT.
https://en.wikipedia.org/wiki/MQTT

Interesting.  What's your own use case?

What does it do that our MCWS protocol can't do?

I understand the logic of it being more efficient than polling, but to poll something once a second, or even ten times a second, doesn't seem like a heavy use for any modern computer.  Bandwidth also seems plentiful.

We implemented Z-Wave a long time ago for an IoT project (Engen), but it was a lot of work for little return.  Not efficient.
Logged

mattkhan

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 4231
Re: Publish / Subscribe for changes?
« Reply #2 on: August 26, 2024, 08:17:32 am »

A push mechanism is less resource intensive and more responsive than polling. Polling frequently enough to be responsive bloats MC logs massively (which is annoying when you are looking for something else) and it scales poorly as it's not unusual to need to hit multiple endpoints to get the info you want.

The recently added playing now feature is a limited implementation of a push protocol (as it pushes some data to a file and/or can call an exe).

It would be better if MC just supported something like a websocket based broadcast mechanism. The impression I have from previous posts is that MC's http server is custom built so all such features have to be written yourself (as opposed to just using some lib that has already implemented support for such protocols).

All this is much the same as raised long ago, eg from 2012 https://yabb.jriver.com/interact/index.php?topic=73730.0

Logged

masterjoe

  • World Citizen
  • ***
  • Posts: 235
Re: Publish / Subscribe for changes?
« Reply #3 on: August 26, 2024, 08:30:20 am »

My use case is to monitor where I am currently in MC and display additional info on demand using my own QT renderings on different decives implemented using my dedicated automation application.
(File selection status, playback position etc.)

Polling in my experience is *always* less efficient when changes are not happening too often. You re wasting your energy / time with polling. Also there is a latency regarding the data update rate you are after which can be removed when you use push messaging (like MQTT does).

An MQTT broker can distribute the data to *multiple* subscribed clients in the moment of the event.

E.g. when MC automatically imports new files you can't find these easily with polling and you waste lots of time doing so. Because it might not happen that often. However *when* it happens you could immediately and efficiently react to these new files, e.g. add additional data using your own scraper etc.
I could think of an (optional) additional parameter which can be added to each pushable MCWS function: AddMQTTTopic=name where "name" is the name of the MQTT topic to be created. With a dedicated RemoveMQTTTopic function you could remove such topics again. This way you can fully pre-configure the topics that shall push messages on any change of the configured topic.

MQTT is an industry standard for IoT and used in many commercial products. There are many great libraries for MQTT support out there (eg. https://mosquitto.org/api/files/mosquitto-h.html)

Subscribe / Publish is something used everywhere *instead* of polling due to its flexibility, multi client ability and efficiency.

Maybe it's feasible to subscribe to MC changes using the *same* MCWS data structures and results but preconfigure topics (with the same parameters a MCWS call allows) that can be used with MQTT. Only when there is a CHANGE in data then a message gets out.

MQTT allows automation integration in other automation systems that support MQTT.

So it would be pretty nice :)
Logged

JimH

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 72443
  • Where did I put my teeth?
Re: Publish / Subscribe for changes?
« Reply #4 on: August 26, 2024, 08:53:08 am »

A push mechanism is less resource intensive and more responsive than polling. Polling frequently enough to be responsive bloats MC logs massively (which is annoying when you are looking for something else) and it scales poorly as it's not unusual to need to hit multiple endpoints to get the info you want.
Can you quantify this with any performance stats? 
Logged

lepa

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 2033
Re: Publish / Subscribe for changes?
« Reply #5 on: August 26, 2024, 10:07:02 am »

It depends. To be able to react almost real time you have to poll frequently. If you poll once per second and get useful data about every minute the efficiency ratio will be 1/60

+1 for push
Logged

mattkhan

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 4231
Re: Publish / Subscribe for changes?
« Reply #6 on: August 26, 2024, 10:11:11 am »

Can you quantify this with any performance stats?
use case is building some custom remote, e.g. the homeassistant integration

https://www.nngroup.com/articles/response-times-3-important-limits/

implies you'd need it to poll at least 10 times per second to maintain a fluid interface

assume you're just playing some music and average track length is 3mins then it means a polling solution has made 1800 requests and processed 1800 responses (per track) instead of just receiving a single payload, even if you go with 1 per second and accept your UI can look a bit stuck then that's still 180 each way vs 1

for an interactive remote control, you can mitigate the effect on responsiveness by adopting a pattern where you have a variable refresh rate (more frequent after you issue commands in expectation of an update and then slow down after). Cost is increased complexity in your own software.

In either case, it's a fairly constant load which means both devices involved are likely to stay in a higher power state than they'd otherwise need to be in so that's rather wasteful and inefficient.

I don't think it's possible to quantify anything beyond that without using particular use cases as an example & I don't have that data, fundamentally polling is always the weaker solution and it gets replaced as soon as you want something more efficient/responsive
Logged

JimH

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 72443
  • Where did I put my teeth?
Re: Publish / Subscribe for changes?
« Reply #7 on: August 26, 2024, 10:24:03 am »

How many instructions can a modern CPU execute in a second?

I don't really know.  I stopped paying attention when it passed a million per second.

Unless you can support your ideas with numbers, it's not going to go anywhere.
Logged

masterjoe

  • World Citizen
  • ***
  • Posts: 235
Re: Publish / Subscribe for changes?
« Reply #8 on: August 26, 2024, 10:35:11 am »

As already said:

- pushing is more immediate!
- multiple clients can listen and get the same event / data
- pushing is used everywhere to save ENERGY
- weak devices don't like active polling with high frequencies!
- MQTT is used by IoT as an industry standard
- integration with other MQTT based automation is possible

Why is it required to tell you more?
These are really strong arguments!
Logged

mattkhan

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 4231
Re: Publish / Subscribe for changes?
« Reply #9 on: August 26, 2024, 10:37:00 am »

How many instructions can a modern CPU execute in a second?

I don't really know.  I stopped paying attention when it passed a million per second.

Unless you can support your ideas with numbers, it's not going to go anywhere.
this is rather reductive argument and shows a fundamental misunderstanding of the problem, to summarise

1) it's about responsiveness as delay = negative user experience
2) it's about efficiency (and hence power usage) = it's been rather hot recently and using software that chooses to waste energy because it's inefficient is a bad thing
3) such client software doesn't just talk to MC = it can have a negative impact on that software and it's overall responsiveness
4) MC has already chosen to implement in MC33 a push api (via the playing now plugin), it's just that it's a totally non standard and bespoke solution with extremely limited capabilities (so if you didn't think a push api was worthwhile, why implement it in this v release?)
Logged

lepa

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 2033
Re: Publish / Subscribe for changes?
« Reply #10 on: August 26, 2024, 10:46:23 am »

Mobile's battery saving for remotes... but yeah, seems that this is not going anywhere anyway
Logged

masterjoe

  • World Citizen
  • ***
  • Posts: 235
Re: Publish / Subscribe for changes?
« Reply #11 on: August 26, 2024, 10:49:18 am »

But why? Polling is 1990ies style at best.

And for low powered devices you REALLY want to do push messages, not polling EVER.
Or you are out for this use case!

Can't get it why something so straight-forward is not understood. Really really strange.
Logged

JimH

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 72443
  • Where did I put my teeth?
Re: Publish / Subscribe for changes?
« Reply #12 on: August 26, 2024, 11:09:18 am »

And for low powered devices you REALLY want to do push messages, not polling EVER.
Or you are out for this use case!

Can't get it why something so straight-forward is not understood. Really really strange.
I just don't see a clear win for this, given how much time it would take to do. 

Real measurements might convince me otherwise.

I do understand your logic.

Logged

zybex

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 2619
Re: Publish / Subscribe for changes?
« Reply #13 on: August 26, 2024, 02:42:53 pm »

Polling is unreliable. Events can easily be missed if they happen between two polling cycles; e.g., a user might click NEXT twice but the polling process only sees the last state, not the previous one. This kind of race condition was solved by event-driven systems long ago.

Efficiency is really important as well; imagine 5 displays spread around the house, periodically polling the MC Server to update themselves. The code may need to call into MC once per second to refresh the display - that's 5 x 86400 = 432000 MCWS calls that the server needs to process per day, with corresponding network traffic, memory allocations, database queries and whatnot. All this activity may also prevent the disks and other subsystems on the Server and clients from going to sleep. With an event broadcast system, the server simply sends a packet whenever something happens, and the clients/scripts only need to wait and react to the events.

One other advantage of MQTT is that is can queue incoming events/messages, so you can safely reboot/powerdown a client without worrying about losing important events - they'll be queued for later retrieval if needed. This allows development of scripts/tools that are otherwise not reliable with polling.

Logged

JimH

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 72443
  • Where did I put my teeth?
Re: Publish / Subscribe for changes?
« Reply #14 on: August 26, 2024, 03:38:54 pm »

When I read the topic on Wikipedia, it said there was no queuing, in spite of the name.

I'm not convinced that the numbers you're talking about would put any stress on a typical system.  Imagine what has to happen just to show a second of a movie.

Computers like to be busy.
Logged

zybex

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 2619
Re: Publish / Subscribe for changes?
« Reply #15 on: August 26, 2024, 04:03:58 pm »

"What is message expiry in MQTT v5?"
This interval establishes the duration the broker preserves the PUBLISH message for subscribers that match the topic but are currently offline. If the interval isn't defined, the broker must indefinitely hold the message for matching subscribers, yet disconnected subscribers.

If polling uses 0.1% of CPU and event-pushing uses 0.001%, that's still an 100x efficiency gain. Still, the reliability of events is the more important aspect here, I think. Performance is a non-issue as you said.
Logged

Awesome Donkey

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 7810
  • Autumn shade...
Re: Publish / Subscribe for changes?
« Reply #16 on: August 26, 2024, 04:05:01 pm »

The only true way to see if there's a big enough benefit to going push in MC is to implement a simple version of it and compare to the polling method. The problem is, implementing it (to get a minimal version working) even as a test is still development time and by the time already invested, even for a test, it's probably enough you might as well go all the way and maybe have it both ways if possible.

I think the best way is to show examples of implementations of polling vs push methods (likely best within the same software if possible so it can directly compared) with hard numbers. Is there anything similar enough to MC that uses both and can be measured? That'd likely be helpful in your pitches here, IMO.
Logged
I don't work for JRiver... I help keep the forums safe from "male enhancements" and other sources of sketchy pharmaceuticals.

Windows 11 24H2 Update 64-bit + Ubuntu 24.10 Oracular Oriole 64-bit | Windows 11 24H2 Update 64-bit (Intel N305 Fanless NUC 16GB RAM/500GB M.2 NVMe SSD)
JRiver Media Center 33 (Windows + Linux) | iFi ZEN DAC 3 | JBL 306P MkII Studio Monitors | Audio-Technica ATH-M50x Headphones

Awesome Donkey

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 7810
  • Autumn shade...
Re: Publish / Subscribe for changes?
« Reply #17 on: August 26, 2024, 04:08:42 pm »

If polling uses 0.1% of CPU and event-pushing uses 0.001%, that's still an 100x efficiency gain. Still, the reliability of events is the more important aspect here, I think. Performance is a non-issue as you said.

To be honest, I don't really think stating efficiency is going to help here with the argument because most people don't care about that, they care about speed (performance) and whether if it works or not. Efficiency would be looked at as a "side effect".

That's how I see it at least.
Logged
I don't work for JRiver... I help keep the forums safe from "male enhancements" and other sources of sketchy pharmaceuticals.

Windows 11 24H2 Update 64-bit + Ubuntu 24.10 Oracular Oriole 64-bit | Windows 11 24H2 Update 64-bit (Intel N305 Fanless NUC 16GB RAM/500GB M.2 NVMe SSD)
JRiver Media Center 33 (Windows + Linux) | iFi ZEN DAC 3 | JBL 306P MkII Studio Monitors | Audio-Technica ATH-M50x Headphones

zybex

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 2619
Re: Publish / Subscribe for changes?
« Reply #18 on: August 26, 2024, 04:10:34 pm »

MC could start by adding a simple MQTT Client lib to push the event messages, and it would be up to the users to install a suitable MQTT Server (Broker).

Plenty of libs out there for both MQTT server and client, including C++:
https://github.com/redboltz/mqtt_cpp
https://github.com/eclipse/paho.mqtt.cpp
https://github.com/hobbyquaker/awesome-mqtt?tab=readme-ov-file#clients
Logged

Awesome Donkey

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 7810
  • Autumn shade...
Re: Publish / Subscribe for changes?
« Reply #19 on: August 26, 2024, 04:14:51 pm »

To be fair, I know very little of MQTT (just a little in passing, I'm not really a fan of IoT stuff) so you'd have to forgive my ignorance on the subject with what I'm about to say and ask. Why not some sort of MCWS to MQTT wrapper or translation? I see some of the MQTT benefits posted above, but what actually would be useful for MC? And what are the downsides of using MQTT (in regards to MC)?

P.S. Yes, I know the first question about a MCWS to MQTT wrapper/translation is kinda dumb and realize the complexity/overhead to it and doesn't actually solve anything, but I wanted to ask regardless.
Logged
I don't work for JRiver... I help keep the forums safe from "male enhancements" and other sources of sketchy pharmaceuticals.

Windows 11 24H2 Update 64-bit + Ubuntu 24.10 Oracular Oriole 64-bit | Windows 11 24H2 Update 64-bit (Intel N305 Fanless NUC 16GB RAM/500GB M.2 NVMe SSD)
JRiver Media Center 33 (Windows + Linux) | iFi ZEN DAC 3 | JBL 306P MkII Studio Monitors | Audio-Technica ATH-M50x Headphones

mattkhan

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 4231
Re: Publish / Subscribe for changes?
« Reply #20 on: August 26, 2024, 04:19:12 pm »

It's a pointless discussion as the reasons for an event based system have been given and are well known to any developer of non trivial software. A simple version of such a mechanism has even been added to mc33. If jriver don't want to prioritise building a proper implementation then I think that's unfortunate but that's their prerogative.
Logged

mattkhan

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 4231
Re: Publish / Subscribe for changes?
« Reply #21 on: August 26, 2024, 04:21:41 pm »

Just to note that mqtt would be a pretty heavyweight thing to add imo and I would think that would limit the potential audience a fair bit
Logged

zybex

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 2619
Re: Publish / Subscribe for changes?
« Reply #22 on: August 26, 2024, 04:22:32 pm »

MCWS is passive - it waits for a client/script to call and responds with the requested info.

MQTT is active - it sends messages to registered clients/scripts whenever something happens (events). Like, "App Started", "Track Changed", "Coffee Exhausted", etc. The receiving app can then use MCWS to get additional info in response to this initial event notification. This allows response times in the milliseconds, and the reliability is far superior as ALL events can be caught and handled by the receiving app(s).

A wrapper in this context doesn't make sense as the event message is usually very simple.

Downsides of MQTT: implementation time and effort for the benefit of just a few users. It doesn't translate into more sales. However, these same users may then use this feature to develop plugins and tools for other non-tech users to enjoy, thus growing the ecosystem.
Logged

JimH

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 72443
  • Where did I put my teeth?
Re: Publish / Subscribe for changes?
« Reply #23 on: August 26, 2024, 04:49:32 pm »

I get that it's an elegant solution and ours only just works.

A few apps have been built on our current framework and they seem to work pretty well.

But the world is full of great ideas.  It takes time and money to implement them.

As a wild guess, it would take us at least a month to get the first cut at an implementation, and another month to fix bugs and finish.  And then we'd have to support programmers using it.

Just so we don't get called quiche eaters?




Logged

Awesome Donkey

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 7810
  • Autumn shade...
Re: Publish / Subscribe for changes?
« Reply #24 on: August 26, 2024, 05:00:18 pm »

However, these same users may then use this feature to develop plugins and tools for other non-tech users to enjoy, thus growing the ecosystem.

Like what? I'm wondering if there's a type of unique plugin that could be created with a MQTT implementation versus doing a MCWS plugin.
Logged
I don't work for JRiver... I help keep the forums safe from "male enhancements" and other sources of sketchy pharmaceuticals.

Windows 11 24H2 Update 64-bit + Ubuntu 24.10 Oracular Oriole 64-bit | Windows 11 24H2 Update 64-bit (Intel N305 Fanless NUC 16GB RAM/500GB M.2 NVMe SSD)
JRiver Media Center 33 (Windows + Linux) | iFi ZEN DAC 3 | JBL 306P MkII Studio Monitors | Audio-Technica ATH-M50x Headphones

mwillems

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 5234
  • "Linux Merit Badge" Recipient
Re: Publish / Subscribe for changes?
« Reply #25 on: August 26, 2024, 05:35:50 pm »

Like what? I'm wondering if there's a type of unique plugin that could be created with a MQTT implementation versus doing a MCWS plugin.

So the playing now popup is a great example of the kind of app someone could have written for JRiver.  I actually wrote a playing now popup for Linux using MCWS for polling a few years back.  I published an early version on the forums that used OS notifications instead of popping up a window, but I was working on an actual pop window to make it nicer.  I had it working, but I ultimately abandoned it in part because the overhead was too high.  It used more CPU than I wanted, and I certainly could have optimized it better, but the killer was that the MCWS calls seemed to slow down MC itself when I used it (the MC UI was noticeably less responsive and more prone to hang when my plugin was running), and I couldn't figure out how to tune that away without introducing too much delay into the popup.  I suspect that it's because I had to make a bunch of different calls on every poll and poll very frequently to get the info/granularity I wanted.

Granted I'm not a professional software dev, so someone better at this kind of thing than I am might've been able to do it better, but it's a real life example of someone trying to make a plugin that was potentially stymied by the polling. 
Logged

masterjoe

  • World Citizen
  • ***
  • Posts: 235
Re: Publish / Subscribe for changes?
« Reply #26 on: August 27, 2024, 01:02:48 am »

So the playing now popup is a great example of the kind of app someone could have written for JRiver.  I actually wrote a playing now popup for Linux using MCWS for polling a few years back.  I published an early version on the forums that used OS notifications instead of popping up a window, but I was working on an actual pop window to make it nicer.  I had it working, but I ultimately abandoned it in part because the overhead was too high.  It used more CPU than I wanted, and I certainly could have optimized it better, but the killer was that the MCWS calls seemed to slow down MC itself when I used it (the MC UI was noticeably less responsive and more prone to hang when my plugin was running), and I couldn't figure out how to tune that away without introducing too much delay into the popup.  I suspect that it's because I had to make a bunch of different calls on every poll and poll very frequently to get the info/granularity I wanted.

Granted I'm not a professional software dev, so someone better at this kind of thing than I am might've been able to do it better, but it's a real life example of someone trying to make a plugin that was potentially stymied by the polling.


THIS is a perfect example where a push mechanism would not only reduce the overhead so much that it actually works but also make it truely event driven and reliable.

Polling is actually sampling with a given frequency. Like all sampling it may and will miss data!! You have the Nyquist-Shannon sampling theorem that kicks in here. Look it up!

So therefore polling is NOT a good solution if you want events to drive things. Why are so many apps event driven? Because it's the best way to do things reliably and efficiently for many use cases.

If you don't like MQTT or think its oversized (it works for IoT little devices, so...). Fine. A TCP connection that sends data only when there is a change would be easy!
A push mechanism could simply use a TCP connection, where you first send your configuration for what changes you like to be returned and then you receive changes on this connection when they happen.
Of course you would have to allow multiple such connections for multiple event / client configurations.

Seems doable, no?
Logged

mattkhan

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 4231
Re: Publish / Subscribe for changes?
« Reply #27 on: August 27, 2024, 01:36:31 am »

my point about MQTT is that it means the user has to have an MQTT broker already running in order to use it. For me, this seems an unnecessary barrier for many (e.g. the custom remote control I mentioned) use cases. A more primitive (e.g. simple TCP based subscription) mechanism can be adapted to publish onto an MQTT bus for those who need it but you can't go the other way.
Logged

masterjoe

  • World Citizen
  • ***
  • Posts: 235
Re: Publish / Subscribe for changes?
« Reply #28 on: August 27, 2024, 02:45:14 am »

my point about MQTT is that it means the user has to have an MQTT broker already running in order to use it. For me, this seems an unnecessary barrier for many (e.g. the custom remote control I mentioned) use cases. A more primitive (e.g. simple TCP based subscription) mechanism can be adapted to publish onto an MQTT bus for those who need it but you can't go the other way.

You could also provide an integrated MQTT broker (turned on via option) inside of MC. So then there would be no requirement for setting up a broker yourself.

But sure, having a plain TCP based event / push mechanism is better than having none :)
Logged

mattkhan

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 4231
Re: Publish / Subscribe for changes?
« Reply #29 on: August 27, 2024, 03:23:45 am »

I would prefer something minimal from MC that can be built upon easily externally rather than integrating such things into MC itself. This has the benefit of reducing the effort required to deliver such a thing too (which seems the key concern)
Logged

JimH

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 72443
  • Where did I put my teeth?
Re: Publish / Subscribe for changes?
« Reply #30 on: August 27, 2024, 05:27:22 pm »

After thinking about this, would it help if we published a few things?  Track change, time remaining, and so on?

Maybe requests to subscribe or unsubscribe could be made to Library Server.
Logged

Striker

  • World Citizen
  • ***
  • Posts: 176
Re: Publish / Subscribe for changes?
« Reply #31 on: August 27, 2024, 08:29:17 pm »

The original PlayingNow plugin did almost everything I wanted. 

I didn't use it for a popup.  I used to to create a user configured text file with track state and playback info and then to call a user configured program after writing that file and then update screens I have all over my house.

The new PlayingNow popup has potential to do everything most of us would want.

It is event driven.

For now though, the new PlayingNow popup is limited in what events it reports on, meaning MCWS polling would still be required.

Hopefully the new PlayingNow plugin will evolve enough to be the only thing I would need to get playback info and state from MC.

For now... I am using MCWS polling to get the info I need, running it every second.  It is slowing down my PC?  Not in any way that I can tell...  That once per second works for me, it's responsive enough for me. 

I finally gave up the original PlayingNow plug in so I could use the MC 64bit version.

I have little knowledge of MQQT... does it required an internet connection?  That would be a total deal killer for me.  Varying webpages say it does and other say it does not.  Still MQQT seems like overkill, the original PlayingNow plugin was so easy to use without any additional equipment or connections.

My company used IBM MQ-Series for many applications and it was rock solid and reliable with guaranteed delivery.  Not sure if that is related to MQQT.  Still it required the MQ-Series app to be running to manage messaging.. again totally overkill for MediaCenter.

I have an extensive home automation system in my home... mostly all home grown and one FEATURE very important to me, is none of it requires an internet connection.
Logged

masterjoe

  • World Citizen
  • ***
  • Posts: 235
Re: Publish / Subscribe for changes?
« Reply #32 on: August 28, 2024, 12:57:10 am »

After thinking about this, would it help if we published a few things?  Track change, time remaining, and so on?

Maybe requests to subscribe or unsubscribe could be made to Library Server.

Great! Yes, that would really help! It would be a start!

I see it this way: MCWS is to query things when required / on demand.
The push mechanism is to get change events when they happen - without knowing when they happen.

I would like to have the following events for subscribe  / publish as a starter:

(1) Playback Info (-> when it changes!)
-> would define mutiple events for playback: started / playing / paused / stopped / position changed / finished etc. so that each can be gotten individually if required;
only "playing" is pushing updates every second (with time info) started, paused, stopped, finished only happen when they happen!; "position changed" shall indicate when you skip forward or backward or change the playback position in other ways and deliver the new and previous playback position
-> should include at least the file key so that the file can be identified using MCWS if required; optional fields could come along with the event -> see last paragraph below

(2) Current Position / Selection Change - especially for Theater View
-> should publish the currently focused / selected file(s) in any view - but especially useful for Theater View
-> in Theater View when you move the selection box cursor then if the file (key) under the cursor changes such an event shall occur
-> when NO file is below the selection box cursor then an event shall occur with a file key of -1 to indicate that the focus is no longer on any file
-> the virtual keyboard, the main menu and other menu items would produce a file key of -1
-> of course only when the focused file key changes an event shall be sent; so if the key is already -1 then no further events of that key will be sent afterwards until the next change of the file key occurs

If desired / optionally database fields belonging to a file key should also be sent with the event message.
So that a query using MCWS is not required at all as the desired info comes along with the event. This is the same as the "Fields=..." option in MCWS.
But providing more fields should NOT automatically happen. It would be configured during the subscription to an event which transports a file key.

Thanks :)
Logged

masterjoe

  • World Citizen
  • ***
  • Posts: 235
Re: Publish / Subscribe for changes?
« Reply #33 on: August 28, 2024, 12:58:41 am »

The original PlayingNow plugin did almost everything I wanted. 

I didn't use it for a popup.  I used to to create a user configured text file with track state and playback info and then to call a user configured program after writing that file and then update screens I have all over my house.

The new PlayingNow popup has potential to do everything most of us would want.

It is event driven.

For now though, the new PlayingNow popup is limited in what events it reports on, meaning MCWS polling would still be required.

Hopefully the new PlayingNow plugin will evolve enough to be the only thing I would need to get playback info and state from MC.

For now... I am using MCWS polling to get the info I need, running it every second.  It is slowing down my PC?  Not in any way that I can tell...  That once per second works for me, it's responsive enough for me. 

I finally gave up the original PlayingNow plug in so I could use the MC 64bit version.

I have little knowledge of MQQT... does it required an internet connection?  That would be a total deal killer for me.  Varying webpages say it does and other say it does not.  Still MQQT seems like overkill, the original PlayingNow plugin was so easy to use without any additional equipment or connections.

My company used IBM MQ-Series for many applications and it was rock solid and reliable with guaranteed delivery.  Not sure if that is related to MQQT.  Still it required the MQ-Series app to be running to manage messaging.. again totally overkill for MediaCenter.

I have an extensive home automation system in my home... mostly all home grown and one FEATURE very important to me, is none of it requires an internet connection.

MQTT does not require an internet connection at all. An MQTT broker can run in your local network. MC could be this MQTT broker by itself if desired.
Logged

mattkhan

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 4231
Re: Publish / Subscribe for changes?
« Reply #34 on: August 28, 2024, 09:53:29 am »

thanks for considering it

using the homeassistant integration as an example, we can see it needs to get various bits of info, some of which are just irregular MCWS calls (e.g. the browse tree configuration) but others are trying to get live state

https://github.com/3ll3d00d/jriver_homeassistant/blob/main/custom_components/jriver/coordinator.py#L141

* server health status
* current list of zones (inc active status)
* UI view mode
* playing now info from each zone which provides all the info in https://github.com/3ll3d00d/hamcws/blob/main/hamcws/hamcws.py#L36 (so what's playing, the image url, volume, position, mute state, whether it's live input+ some arbitrary list of fields)

this is mostly aligned to masterjoe's 1) though note that it covers multiple zones

Logged

mattkhan

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 4231
Re: Publish / Subscribe for changes?
« Reply #35 on: August 28, 2024, 09:54:48 am »


-> should publish the currently focused / selected file(s) in any view - but especially useful for Theater View
-> in Theater View when you move the selection box cursor then if the file (key) under the cursor changes such an event shall occur
-> when NO file is below the selection box cursor then an event shall occur with a file key of -1 to indicate that the focus is no longer on any file
-> the virtual keyboard, the main menu and other menu items would produce a file key of -1
-> of course only when the focused file key changes an event shall be sent; so if the key is already -1 then no further events of that key will be sent afterwards until the next change of the file key occurs
presumably you'd want that debounced? like cat sits on the down arrow and scrolls through a list of 10000 entries, you don't want 10k events sent to you
Logged

mattkhan

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 4231
Re: Publish / Subscribe for changes?
« Reply #36 on: August 28, 2024, 09:55:57 am »

btw json strongly preferred for this purpose imv
Logged

cncb

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 3106
Re: Publish / Subscribe for changes?
« Reply #37 on: August 28, 2024, 10:04:11 am »

What about elapsed time?  Would you expect a push every second?  Doesn't seem much different than polling in this case.
Logged
-Craig    MO 4Media remote and player:  Android/TV/Auto | iOS | Windows 10/UWP

mattkhan

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 4231
Re: Publish / Subscribe for changes?
« Reply #38 on: August 28, 2024, 10:16:47 am »

If I were designing this, I would probably publish an update at very low frequency (eg every 10s) containing v minimal info ( position, zone, file key) and allow the app to assume progression unless playback state changes.

Alternatively you let the app tell you how frequently to update (and assume a long interval or that you won't get such messages at all if not set)
Logged

Hendrik

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 10940
Re: Publish / Subscribe for changes?
« Reply #39 on: August 28, 2024, 10:47:37 am »

What about elapsed time?  Would you expect a push every second?  Doesn't seem much different than polling in this case.

Assuming it runs at real-time, no need to really update a clock moving forward that often, as long as state changes are communicated when this situation changes.
Logged
~ nevcairiel
~ Author of LAV Filters

masterjoe

  • World Citizen
  • ***
  • Posts: 235
Re: Publish / Subscribe for changes?
« Reply #40 on: August 29, 2024, 03:19:45 am »

presumably you'd want that debounced? like cat sits on the down arrow and scrolls through a list of 10000 entries, you don't want 10k events sent to you

There could / should be an option per event for throttling. So that events are only fired when no more changes occur for n miliseconds (configurable per event).
Throttling must still ensure that the LAST change event ALWAYS occurs but not the in-betweens (which occur faster than the throttling timeout). So when fast scrolling appears the next change will only happen when there is no more change for say 200ms (the throttling timeout, which can be 0 = off).

In my use case I do not scroll huge lists at all...
Logged

masterjoe

  • World Citizen
  • ***
  • Posts: 235
Re: Publish / Subscribe for changes?
« Reply #41 on: August 29, 2024, 03:25:55 am »

If I were designing this, I would probably publish an update at very low frequency (eg every 10s) containing v minimal info ( position, zone, file key) and allow the app to assume progression unless playback state changes.

Alternatively you let the app tell you how frequently to update (and assume a long interval or that you won't get such messages at all if not set)

I would want to configure the throttle time per event and have a default of 0.
With a throttling timeout you can define yourself how quick you want changes to be detected. 0 means: get every change as event!
100 means: send a change event for the current value if this value has changed but has not changed within 100ms.

With this you have maximum flexibility.
Logged

masterjoe

  • World Citizen
  • ***
  • Posts: 235
Re: Publish / Subscribe for changes?
« Reply #42 on: August 29, 2024, 03:43:52 am »

Assuming it runs at real-time, no need to really update a clock moving forward that often, as long as state changes are communicated when this situation changes.

Actually I do not need a clock update event either.
However the suggested "playing" event could communicate time elapsed / remaining automatically - if you subscribe to it.
Other events do occur unpredictably on change.
Logged

masterjoe

  • World Citizen
  • ***
  • Posts: 235
Re: Publish / Subscribe for changes?
« Reply #43 on: August 29, 2024, 03:50:34 am »

btw json strongly preferred for this purpose imv

I like JSON for the events, too.

When the event transports a file / multiple files it's maybe a good idea to be able to configure similar formats on subscribe as they are possible for Files/Current:
MPL: return MPL playlist
JSON: Return files as JSON array
Serialize: return serialized file array (basically a list of file keys)

Requested DB fields should be optionally includable where possible.
Logged

mattkhan

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 4231
Re: Publish / Subscribe for changes?
« Reply #44 on: August 29, 2024, 09:01:55 am »

I would want to configure the throttle time per event and have a default of 0.
With a throttling timeout you can define yourself how quick you want changes to be detected. 0 means: get every change as event!
100 means: send a change event for the current value if this value has changed but has not changed within 100ms.

With this you have maximum flexibility.
This doesn't make sense for position

the reason to provide a snapshot on a slow interval is just a mechanism to keep in sync in case of loss or other issues, it's just an aide to reliability really
Logged

zybex

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 2619
Re: Publish / Subscribe for changes?
« Reply #45 on: August 29, 2024, 09:52:35 am »

I don't think a PositionChanged event needs to be fired regularly, it's overkill. It's something that increases by 1 every 1 second, that can be done on the client side code. If "time remaining" reaches zero (or -1) on the client and it hasn't yet received a new event for a track change, it can then poll the server to find out what's wrong. There should be a PositionChanged event, but to be fired only when the user manually moves the needle (seek).

In fact, none of the events should have fixed-periodicity or throttling limits (for the cat-on-keyboard scenario, the OS already limits the max key-repeat frequency).

The currentFile/playing/seeking/stopping/pausing/appClosing events are enough for accurate tracking, along with other State-changed events triggered by keypresses, mouse clicks, or other manual or automated functions.

AppStarted is trickier as there's no client connection at that point to send it. It's up to a client to determine (using polling :)) if the socket is open again for reconnection.
Logged

zybex

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 2619
Re: Publish / Subscribe for changes?
« Reply #46 on: August 29, 2024, 10:48:39 am »

Proposed protocol, something like below:
red = messages from client to MC
blue = messages from MC to client
green = comments
< and > are not part of the message, it's just to indicate direction.

< HELLO { MCVersion: "33.0.15", APILevel: 1 }                  // greeting message from server on connection
> HI ALEXA!                                                    // client sends invalid command
< What? Unknown command                                        // standard error message             
> Nevermind...
< What? Unknown command
// client registers for all events, server says OK and immediately sends the current PLAYSTATE (could also send other events for current mode/view/state)
> SUBSCRIBE { event: "*", fields: "key" }                                           
< OK
< EVENT { seq: 1, time: 1724945366, event: PLAYSTATE, state: PLAYING, position: 213, Key: 12345 }

// client re-registers for PLAYSTATE events with a specific field list, server says OK and resends the current PLAYSTATE
> SUBSCRIBE { event: "PLAYSTATE", fields: ["Key", "Name", "Track #", "Disc #", "Album", "Year", "Artist", "Album Artist", "Duration"] }
< OK
< EVENT { seq: 2, time: 1724945366, event: PLAYSTATE, state: PLAYING, position: 214, Key: 12345, Name: xxxx, "Track#": 1, Duration: 280, .... }
// event sent when track finishes
< EVENT { seq: 3, time: 1724945433, event: PLAYSTATE, state: STOPPED, position: 280, Key: 12345, Name: xxxx, "Track#": 1, Duration: 280, .... }
// next track starts
< EVENT { seq: 4, time: 1724945433, event: PLAYSTATE, state: PLAYING, position: 0, Key: 3453, Name: xxxx, "Track#": 2, Duration: 366, .... }
// user changes the volume
< EVENT { seq: 5, time: 1724945445, event: VOLUME, value: 100 }
// user pauses playing
< EVENT { seq: 6, time: 1724945623, event: PLAYSTATE, state: PAUSED, position: 190, Key: 3453, Name: xxxx, "Track#": 2, Duration: 366, .... }
// user switches to Theater Mode
< EVENT { seq: 7, time: 1724945630, event: VIEWMODE, mode: "THEATER" }
// user starts a track
< EVENT { seq: 8, time: 1724945787, event: PLAYSTATE, state: PLAYING, position: 0, Key: 832343, Name: xxxx, "Track#": 3, Duration: 324,  .... }
// user seeks (position change)
< EVENT { seq: 9, time: 1724945809, event: PLAYSTATE, state: PLAYING, position: 120, Key: 832343, Name: xxxx, "Track#": 3, Duration: 324, .... }

...

Logged

JimH

  • Administrator
  • Citizen of the Universe
  • *****
  • Posts: 72443
  • Where did I put my teeth?
Re: Publish / Subscribe for changes?
« Reply #47 on: August 29, 2024, 10:57:50 am »

HELLO zybex
Want a job?

Logged

zybex

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 2619
Re: Publish / Subscribe for changes?
« Reply #48 on: August 29, 2024, 11:20:26 am »

What? Unknown command

;D

Need to think about it... Not before January.
Logged

mattkhan

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 4231
Re: Publish / Subscribe for changes?
« Reply #49 on: October 18, 2024, 02:21:43 am »

another example of having to workaround the lack of this capability in https://yabb.jriver.com/interact/index.php/topic,137808.msg969862.html#msg969862

same story really (repeatedly querying for rarely changing data is inefficient)
Logged
Pages: [1]   Go Up