Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[zybex]

MC Team, can you please investigate?

I tested MCWS speed using the new file Share feature, versus serving the same file on a standard Apache server.
The hardware, software and network config is exactly the same - Apache is running on the same PC as MC.
Jmone also ran a speed test to my server from his Australian home, and I simulated connections from other locations using a VPN.
I have 500/25 cable internet; this corresponds to a max upload speed of about 3 MB/sec.

Results in MB/sec below. It's clear there's a problem with MC's web server speed, and it gets worse with latency. Even over LAN the results are bad.

(note: VPN latencies are twice normal, since both the Ping packet and the Reply have to travel both ways. Hence the large Ping difference from the two Australia tests)

[jmone]

I also see a major speed difference between MC's vs Nextcloud.  MC is much slower with the links I've posted in the other thread that can be used for testing.

[Hendrik]

I can download from my MC instance over local network at 70-80MB/s (thats ~650 Mbps), and when testing from an internet server downloading from my local connection, also maxes out my upload. I don't have a very high latency case to test, but at least I don't see any of your behavior on a first glance.

I will do further testing next week, the send buffer size might need re-evaluating, where too low values would interact badly with latency.

[jmone]

I can confirm what Hendrik sees as well for pure LAN clients.  I did some tests on my local network using wget on a 1GB file asking MC for a file using the "share" link(s):

Benchmark Specs (this was all done on the local machine where MC Server is running) = As good as it gets
- Local IP Address: 373 MB/sec (eg full disk speed)
- WAN IP Address: 95 MB/sec
- FQDN Address: 82 MB/sec (had to use the wget "--no-check-certificate" option for some reason)

From my server to my PC over a 10 GB/sec fibre connection = Should also be good as they are on the same LAN and all connections are 10 GB fibre but it may have to bounce through a set of switches or even U turn in the router (I did not see any traffic go external)
- Local IP Address: 188 MB/sec
- WAN IP Address: 95 MB/sec
- FQDN Address: 73 81MB/sec (had to use the wget "--no-check-certificate" option for some reason)

So MC's Shared Links are plenty fast enough on the LAN.

[zybex]

I can download from my MC instance over local network at 70-80MB/s (thats ~650 Mbps), and when testing from an internet server downloading from my local connection, also maxes out my upload. I don't have a very high latency case to test, but at least I don't see any of your behavior on a first glance.

You can't just compare your MC speeds with my MC speeds - it's running on different hardware/software/network, so of course there will be differences. The point of my testing was precisely to change a single thing (the web server), so that results can be compared. You need to do the same if you want to observe the same results.

I tested this with MC/Apache running on a low power HTPC: Quad Core Intel Pentium @ 2.2 GHz, with 10W TDP. Of course there's better, but the point remains: Apache is much faster on the same hardware. Even on this low power PC I see MC using just 25% CPU during these tests, and Apache using 35%. So it's not a CPU bottleneck.

The main problem is not LAN speed - having 40 or 80 MB/sec is irrelevant for streaming over LAN, both are fine. But 200 KB/sec instead of 2 MB/sec for remote streaming? That's not good.

Note that all numbers are in MByte/sec, not Mbit/sec. MB = MByte, Mb = Mbit.

[zybex]

I installed MC Server and Apache on my gaming machine and repeated the tests.

Conclusion: same results - see attached:
- On localhost, MC is actually 30% faster than Apache, at an amazing 1.2 GB/sec. This proves that MC can be fast.
- over wired LAN, Apache almost saturates the link (98MBps ~ 800 Mbps), but MC already drops to 580 Mbps
- over wifi LAN, Apache again saturates the wifi link at over 600 Mbps, but MC can't even reach 200 Mbps
- as latency increases... speed goes dramatically down, up to 90% worse than Apache. This is not a small difference.

Since the CPU is not to blame, this seems to be related to TCP stack implementation on MC: insufficient buffer sizes, TCP scaling window not growing, delayed ACK not enabled, or something like that. For the remote sharing feature, it would be nice to find the culprit. Perhaps MC was just optimized for Wired LAN and needs some tweaks to handle latency better. This will also likely benefit Cloudplay uploads and improve the Panel/JRemote/etc experience.

Test notes:
- Values are in MBytes/sec.
- Tests were done with source files on a local M.2 SSD (3GB/sec read speed).
- I used "wget -O nul" to perform read-only tests: file is downloaded but not written to disk, so the disk on the test machine is not relevant.
- only "localhost" test was executed on the Server machine; other tests were run on a 2nd PC connected to LAN via Ethernet or Wifi

[lepa]

I'm also getting much higher upload transfer rates when using other server solutions to send file. MC seems to peak ~10mbps for me while using something other I can get ~50mbps where my mobile downlink is possibly the bottleneck

Tested using same HW (PC to phone) and network (fixed to 4G).

[Hendrik]

I already have a pretty good idea what's causing it, but need a good reproduction case as well as look into the history of the current code as well as any cross platform considerations.

[Hendrik]

There are changes in the next build that should hopefully help with this.

[jmone]

Looking forward to checking it out.  Thanks.

[zybex]

Thanks Hendrik!

[lepa]

What a good way to start the week!

[jmone]

The JR team has made some improvements to the speed of the streaming (in an upcoming versioin), how do these links work (coming from Australia over a 50Mb/sec uplink)?
- Video @ 10Mb/sec
- Video @ 15Mb/sec
- Video @ 20Mb/sec
- Video @ 40Mb/sec
- Video @ 80Mb/sec
Thanks
Nathan

[zybex]

Still slow. Can you please move a bit closer, like, to Spain? 

This build is definitely better, I now see the same speeds for MC vs Apache on my tests.
For your links, it's all over the place today, it seems connection to AU is overloaded now. I'll test again tomorrow. I see it fluctuating between 100 KB/sec and 1MB/sec, which is better than before (it never went above 200 KB/sec), but it's too unstable today.

I also see HTTPS being quite a bit slower than HTTP over LAN/wifi (45 vs 75MB/sec), possibly due to MC not using CPU-accelerated encryption. This is only noticeable at high bandwidths and with a weak CPU though, so I don't think it's a concern.

[jmone]

Spain sounds nice!

FYI - Great job on the tweaking by Hendrik!  I too find that MC no longer seems the bottleneck.   From my tests the botteneck is (as expected) a combo of:
- Bandwidth the Client has for download, and
- My current 50Mb/s upload

So currently, it looks like if I'm to host Videos off MC the "sweat" spot will be <=15Mb/s given where we are with Bandwidth here in Oz.  For my brother in the UK.... well he might need to download the videos first to prevent buffering, but I can work with 15Mb/s.

[lepa]

Getting my max bandwidth speed to my phone now I assume same fixes also applies when setup is MC server and MC client on PC over internet

[zybex]

Tested your links now - still won't go above 600KB/s, often slower. Maybe lots of people are watching your trains now

Regarding the Cert error with wget... did you try using the fullchain.pem as public key in MC?

[jmone]

It is peak evening time here in Oz so maybe the Nextflix factor? 

From my router I see I hit a max upload of 42.5Mb/s sometime this arvo so that is pretty maxed out.  Do you know how international bandwidth works, in that:
- It seems to get slower the longer the ping time (I presume it is waiting for ACKs and the whole thing slows down?)
- If I got more upload Bandwith, would I then get better throughput to OS users.... or is it still a crap shoot?
- Is there anything else I can do from my end to get better throughput to OS users.

Re the Cert:  I'm not sure what got spat out of Certify.  I got a PFX file that then extracted a CRT and KEY file that I loaded into MC.

[jmone]

@zybex.... I "presume" that 600KB/s was from MC and my NextCloud links?  If not, could you check to see if one is better than the other?

[zybex]

Yes, identical speed for both. Though your jmone.org is still providing 5-6 MB/sec. Your ISP is starved.

Cert: the CertBot produces cert.pem (containing just the public key) and a fullchain.pem (containing the public key plus all certs up the chain). With cert.pem my MC requires the --no-check-certificate, but with fullchain.pem it just works since all needed certs are there to validate the chain until root.

Unrelated: since you own jmone.org, you could perhaps use mymc.jmone.org instead of dyndns... it would require your domain DNS to be hosted on a service supporting dynamic DNS updates.

[Hendrik]

- It seems to get slower the longer the ping time (I presume it is waiting for ACKs and the whole thing slows down?)

Thats not supposed to happen, its more likely that your ISP has limited overseas capacities.

[jmone]

Thanks Gents.  I'm slowly unwinding my 15yo Hosting setup (kind of like cord cutting), but having to learn all the "new ways" of doing this stuff. 

[jmone]

Thats not supposed to happen, its more likely that your ISP has limited overseas capacities.

I think you are correct.  I've taken a leaf from zybex and did some testing using a VPN:
- No issues making my link out from a Sydney VPN
- All the OS VPN sites were terrible.  The "best" was Singapore where I managed to hit a peak of 10Mb/s but as with all of them it fluctuated badly.
- I'd say the average would be under 5 Mb/sec and many times it almost crawled to a stop.

[jmone]

I've added a 250/100 service from another ISP to test if higher upload speed helps with connection speed and reliability out of Australia to the rest of the world. 

Zybex has been testing this for me (thank!), and unfortunatly it does not look like it makes any real difference.  He can still only get at best a peak <10Mb/s and typically way way less. 

Anyway, if anyone else wants to see how it goes here is the test links again with some higher bit rate material, but I'd be interested if anyone can even get the 15Mb/s version playing without buffering!

- Video @ 10Mb/sec
- Video @ 15Mb/sec
- Video @ 20Mb/sec
- Video @ 40Mb/sec
- Video @ 80Mb/sec
Thanks
Nathan

[JimH]

The first two links don't work for me. 

This site can’t be reached

mymc.dyndns.info took too long to respond.

[JimH]

C:\Users\JimH>tracert mymc.dyndns.info

Tracing route to mymc.dyndns.info [159.196.100.161]
over a maximum of 30 hops:

  1     5 ms     3 ms     3 ms  modem.Home [192.168.0.1]
  2    20 ms    12 ms    12 ms  stpl-dsl-gw15.stpl.qwest.net [207.109.2.15]
  3    12 ms    13 ms    18 ms  stpl-agw1.inet.qwest.net [207.109.3.113]
  4    33 ms    15 ms    29 ms  4.68.38.177
  5    38 ms    22 ms    23 ms  ae-1-3510.edge1.Chicago10.Level3.net [4.69.219.14]
  6    23 ms    23 ms    31 ms  ae-4.r07.chcgil09.us.bb.gin.ntt.net [129.250.8.173]
  7    28 ms    26 ms    24 ms  ae-5.r23.chcgil09.us.bb.gin.ntt.net [129.250.4.213]
  8    66 ms    64 ms    63 ms  ae-1.r24.snjsca04.us.bb.gin.ntt.net [129.250.5.17]
  9    68 ms    63 ms    82 ms  ae-19.r01.snjsca04.us.bb.gin.ntt.net [129.250.3.27]
 10    63 ms    63 ms    63 ms  ce-0-17-0-0.r01.snjsca04.us.ce.gin.ntt.net [128.242.179.34]
 11   218 ms   214 ms   212 ms  be6.core1.equinix-sy1.syd.aussiebb.net [180.150.2.109]
 12   213 ms   211 ms   216 ms  be5.core1.vdc01.syd.aussiebb.net [180.150.1.156]
 13   214 ms   212 ms   218 ms  HundredGigE0-0-0-20.bng3.vdc01.syd.aussiebb.net [180.150.1.133]
 14   218 ms   219 ms   214 ms  159-196-100-161.9fc464.syd.static.aussiebb.net [159.196.100.161]

Trace complete.

[JimH]

It would be interesting for you to play the same file from zybex's server.

[jmone]

I guess what I really need to determine if Packet Loss is an issue on these OS links and hence grinding my effective upload speed to a crawl.  A simple ping shows I normally get no Packet Loss, but in just testing to jriver.com I saw 3%... and given a video file will have lots of packets.... :

Code: [Select]

Ping statistics for 216.14.187.161:
Packets: Sent = 100, Received = 97, Lost = 3 (3% loss),
Approximate round trip times in milli-seconds:
    Minimum = 200ms, Maximum = 251ms, Average = 214ms
Also to Zybex's server

Code: [Select]

Ping statistics for xxx.xxx.xxx.xxx:
    Packets: Sent = 100, Received = 100, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 315ms, Maximum = 382ms, Average = 321ms

[jmone]

...I should say a way of working out what the packet loss is like when downloading from a MC Link (rather than just using Ping).

[jmone]

I'm also playing with Rate Limiting on the Upload port to see if that helps. 

[mattkhan]

...I should say a way of working out what the packet loss is like when downloading from a MC Link (rather than just using Ping).

Use wireshark, you will see your throughout looks like a sawtooth. I looked the other day and it drops like a stone intermittently and then slowly comes back up. It looked like tcp congestion control and slow start to me but I didn't dig into it. Usual thing over a congested wan link without any QoS to prioritise your traffic basically.

[JimH]

I wonder if Amazon S3 has servers in Australia that have their own connections to the rest of the planet.

I suppose we could allow our S3 server to be used.  It would be interesting to set up a test.  But Bob is head-down on another project right now.

[jmone]

Yeah there is AWS in Oz. 

Anyway, I've just changed how I do rate limiting and from my end it looks better if anyone wants to test again?

[mattkhan]

20Mb/s one works fine here, 40Mb/s was too much

chart was I referring to earlier produced by wireshark for the 40Mb attempt

[jmone]

Thanks Matt (appreciate the time to test!),

I happened to be on my PC and could see someone was streaming a file!  Looks like after I changed the rate limiting method at my end we have a much better result.  In my testing earlier in the day, I could see (by using a VPN to UK / US/ EU) that a stream would ramp up, get a goodish stream going around 25Mb/s on average (sometimes peaking into the 40's and sometime dropping down a bit), a bit like what your chart shows.  It seems that you REALLY don't want to drop packets when streaming across links with long latency.  As a comparison, another mate near me in Sydney had no issue in watching a 80Mb/s encoded video in realtime, but the latency between us is only a few ms. 

Anyway, 25Mb/s would work nicely for me . 

[zybex]

It's MUCH better now. Ramps up quickly to over 4 MB/sec, averaging at around 3.5MB/sec.
This is the 40Mbps clip:

nul      100%[======================================>] 152.68M  4.27MB/s    in 42s

2021-02-23 08:57:21 (3.61 MB/s) - 'nul' saved [160101819/160101819]

Still, it took 42 seconds for this 33sec clip to download. This reflects on the playback - it buffers for around 14 seconds before playback starts. 40Mbps is definitely too much.
The 20 Mbps one buffers for just 5 seconds, and then plays fully without pauses.

Nice progress You should have mentioned before that you had QoS enabled!

[jmone]

Hi zybex, thanks for your ongoing testing and patenice of this.  It has been like peeling an onion .... there is always another layer.  What I've found so far is
- To 'cap' speeds to your plan on the Australian NBN (National Broadband Network) they use a "Policer" that drops all packets that exceed your plan.  The ISP's manage the download so you don't see any issues in that direction but Uploads will suffer from dropped packets unless you rate limit your uploads somehow.  I hit that problem a couple of weeks ago with upload to Australian sites.
- You can "fix" this by simply applying QOS and setting the max upload speed to be less than your plan.  One side effect of this was on the Ubiquiti UDM Pro is that it tends to cap your uploads to around 800Mbps when you turn on "Smart Queues".  Others and found a different solution that I had been using which was:
- Placing a switch between your Router (UDM Pro) and the Fibre Port (NBN UNI-D) and apply in the switch port an uplike rate limit at just less than your plan.  This seemed to work great as I was not getting 930/47 on a 1,000/50 plan to Australian sites but my OS upload (as you know) was terrible.  Turns out this idea of using a switch is not great as it too just drops packets which starts to become an issue on long latency links (like to the EU / US for me), so today:
- I removed the switch and went back to just using the UDM Pro's inbuild "Smart Queues" (aka QOS) and set the upload speed at a bit less than the plan I've been testing on (250/100).  In discussion this with the ISP, it was suggested that I may see better performance with a lower upload limit but that will take some trial and error to see where the best sweet spot is for keeping us much raw uplink speed vs avoiding dropped packets.

When Matt was testing my uplink was set to 99Mb/s and in looking at his chart it has some big drops (lost packets?).  I've reduced the uplink speed to 95Mb/s but I'm not sure if that change made it before your test or not.

I'll do some more testing using the VPN and see if I can work out the balance between rate limiting the upload speed and the best actual and consistent throughput. 

I hope the above helps some other poor sod in the future that gets a high speed plan then wonders why their upload is unreliable.

Thanks everyone for testing!

[Hendrik]

- To 'cap' speeds to your plan on the Australian NBN (National Broadband Network) they use a "Policer" that drops all packets that exceed your plan.

If that is whats going on, thats some really crappy way of limiting, and no wonder it causes significant drops over high latency as it requires re-transmits.

[wer]

It's actually a very common practice, because dropping the packets is less processor/memory intensive than implementing QoS or traffic shaping that would throttle more gracefully. And that matters on large access concentrators.

[zybex]

NBN is a wholesaler, and most ISPs are just reselling their service. I think they do that to prevent abuse by the ISP resellers.
Anyway, they're supposed to be overprovisioning, so you should not have to limit your own upload rate:
https://www.zdnet.com/article/nbn-to-skip-tcpip-overprovisioning-for-1gbps-plans/

HOWEVER, when upload is saturated on your home connection it often has a dramatic effect on the download speed, because the ACK packets going out are also being queued and not prioritized. So what you should do, if your router allows it, is prioritize ACK and ICMP packets so that download speeds are not affected when upload is saturated.

[jmone]

The ISPs do shape the download (to avoid the NBN Policer) but physically can not do that for the upload unless you are using their Modem/Router which they have pre configured.  The issue comes with Noobs like me NOT using the ISP provider equipment.  As a FTTP customer, I can just plug straight into one of the NBN Termination device ports (in my case a Gigabit Ethernet Port) so need to do my own shaping or the NBN Shaper kicks in to keep me within the plan..... and hence my throughput drops significantly.  In fact I currently have two active ethernet connections (one is on a 1000/50 plan, the other on a 250/100 plan) to test this whole upload thing.   

If may be different for those on FTTN / FTTC / HFC as you have to use an ADSL or Cable Modem anyway.  Here is a FAQ on it for pfsense for example  - https://whirlpool.net.au/wiki/pfsense_traffic_shaping.

There is some overprovisioning for download speeds (but they have now excluded the 1Gb plans) and that is why I get 275 dl on the 250 plan, but "only" 930 on the 1Gb plan.  IFAIK there is no overprovisioning for upload at all.

I just tested various upload speed settings setting on my UDM pro over a VPN and got similar results with settings from 90-100 Mb/s but when I went above this or turn off "Smart Queues" then I got the poor upload speed.

I'm traveling for work (weird I know) for the next couple of days so will leave it on 95Mbs for now.

[jmone]

The 250/100 service was good here in Oz, I could even stream 80Mb/s encoded video files from MC over 5G just fine.

@Zybex/Matt/Others, do you mind testing one more time?  I've swapped back to the 1000/50 service (with QOS on at 49Mb/s to avoid the limiter) and I would be interested to see how much (if any) impact there with the Upload now halved.  This should be a good test to see what sort of service I'll need to settle on for streaming from MC.

Thanks
Nathan

[zybex]

Set up uncontrolled tests, get random results   
There are likely more people testing at this time, or maybe yourself.

1st test: 300 KB/sec for half the file, then jumped to 1MB/sec. Average 700 KB/sec
2nd test: started good, then dropped to 500, then raised again. Average 1MB/sec
3rd test: peak of 3MB/sec, average of 2MB/sec. Is this the real speed? Who knows.


PS: the ? ? ? smiley doesn't seem to be working.

[jmone]

Thanks - I'm not surprised as I'm moving links over to use this connection from the web host.  This is actually a very good test of why I'd be better with a more symmetrical (and higher) upload speed as even if the videos are only encoded at 20Mb/s, it would only take two simultaneous connections to max my upload on a 50Mb/s link. 

I'm going to swap back to the trial 250/100 connection for now.  I'm waiting for more plans to become available as the current NBN pricing structure in Australia makes anything with higher uploads very expensive. 

[jmone]

Back to 250/100, now waiting for a POI to go live from one ISP to get something like 200/200

[zybex]

Can't wait for FTTH here. Germany is waaaaaay behind on fiber deployment.

You know you could reencode your videos to HEVC+AAC... just 5 Mbps gets you excellent 1080p quality. Even half that is already better than Netflix, IMHO. That allows you to have multiple people streaming without killing your connection.

[jmone]

Yeah, I initially tried to use HEVC (as that is what I encode in for my main Videos).... but I found poor support in Web Browsers
- Chrome:  Plays without Video (black screen)
- Edge: Plays but the video is jerky

Here is the 30sec Sample Clip encoded at 10MB/s but with HEVC + AAC in MP4 in UHD 50fps
- MC Link
- NextCloud Link

I've not tried every possible encoding combo so maybe there is something that works more consistently? 
...so I ended up back at AVC as that just "works"

EDIT: The table on this page https://en.wikipedia.org/wiki/HTML5_video suggests that HEVC is really not supported by most browsers while 264 looks pretty well universally supported.

[jmone]

Can't wait for FTTH here. Germany is waaaaaay behind on fiber deployment.

...and I thought we were way behind!  Only got FTTP a couple of months ago and I had to pay extra to go from FTTC to FTTP.

[Hendrik]

FTTC is pretty standard here by now. Just extending it closer then that is usuay only available in very new constructions where it's planned right into everything, or from select local providers that use it to get an edge over the big nationwide ISPs.

The reasons are of course the old ones. Using existing phone lines for DSL or TV cable lines allows connecting many households without expensive construction work

The good news is that the street infrastructure is already there so if they decide to go with fiber to the home eventually, that part won't need to be build up first.

It's not all bad. I still get 1000/50 for cheap, and they are extending the upload to 100 by the end of the year, or so I am told.