Cloud server performance issue

[slivovitz]

Hi,

We have a setup of multiple WCS servers on Azure. Recently we did load testing to determine maximum load of a server and we came to some unexpected results.

Server details: 16 vCPUs and 64GB of RAM, Azure Das_v4 series
Operating system: Ubuntu 20.04
JDK: openjdk version 12.0.2

Configuration and tuning done according to instructions here and here:
- Java heap memory configured to 32g
- ZGC configured
- logging set to minimum (error) level

Input stream is limited at 1200 kbits.
There is no transcoding.

We tested using WebRTC pulling and managed to get about 400 streams out. As soon as we tried to pull over 400, degraded_streams_percent went to 100%. At the same time CPU percentage was around 20% and there was more then enough free memory:

Does anyone know what could be the cause of this? With the current setup we were hoping to the get 700-800 streams per server.
It's strange that we hit degraded streams when we're are not close to max out on CPU and memory.

I can send our WCS config files and logs via email: logs@flashphoner.com
Let me know what else can be useful.

Thank you

 

[Max]

Good day.
Please add the following settings to flashphoner.properties file on server you test:

rtp_paced_sender=true
rtp_paced_sender_initial_rate=200000
rtp_paced_sender_increase_interval=50
rtp_paced_sender_k_up=0.9

This settings allow mutithreaded encryption (one thread per every playing session), which should help to escape stream degradation.

 

[slivovitz]

That solves it! We are able to run 1000+ streams. Here is a test with steam quality of 1500kbits:

Thanks a lot for the help!