Video freezing after a few hour

T

tonyportersec

Member
Just did a long run testing, after viewing video for a few hour, the video was freezing on both windows chrome and edge on same PC from 17 different rtsp cameras. If i using /rest-api/stream/find_all to check, all the session and rtsp are still connecting, if i stop the session without pubishing again and start session again, it is fine. What shoud i check? Thanks

The configuration:

RTSP cam x 17 ------(RTSP)-------WCS-------(Webrtc)----2 browsers, Chrome (8 cam) and edge (same 8 cam + others 9 cam)
 
Max

Max

Administrator
Staff member
Good day.
This seems like WebRTC issue. Please enable stream distribution optimization on your server:
Code: 
streaming_distributor_subgroup_enabled=true 
streaming_distributor_subgroup_size=50 
streaming_distributor_audio_subgroup_size=500 
streaming_distributor_subgroup_queue_size=300
streaming_distributor_subgroup_queue_max_waiting_time=5000
streaming_distributor_audio_subgroup_queue_size=300
streaming_distributor_audio_subgroup_queue_max_waiting_time=5000
Please also check if channel bandwidth from server to your test PC is good enough to pass 17 streams with at least 1 Mbps bitrate each (as we saw in your test streams). Use iperf over TCP for channel testing as described here.
 
T

tonyportersec

Member
Max said:
Good day.
This seems like WebRTC issue. Please enable stream distribution optimization on your server:
Code: 
streaming_distributor_subgroup_enabled=true
streaming_distributor_subgroup_size=50
streaming_distributor_audio_subgroup_size=500
streaming_distributor_subgroup_queue_size=300
streaming_distributor_subgroup_queue_max_waiting_time=5000
streaming_distributor_audio_subgroup_queue_size=300
streaming_distributor_audio_subgroup_queue_max_waiting_time=5000
Please also check if channel bandwidth from server to your test PC is good enough to pass 17 streams with at least 1 Mbps bitrate each (as we saw in your test streams). Use iperf over TCP for channel testing as described here.
Click to expand...

The bandwidth from server to PC should be ok, as there is only our browser connecting (Still in testing phrase) and If it is not enough, should the problem happen immediately, not after a few hour.

Stream distribution optimization
What actually it is doing? Any side effect?
 
T

tonyportersec

Member
iperf3: interrupt - the client has terminated

C:\Tools\iperf-3.1.3-win64>iperf3.exe -c XXXXXXXXXXX -p 5201 -R
Connecting to host XXXXXXXXXXXX, port 5201
Reverse mode, remote host XXXXXXXXXXX is sending
[ 4] local XXXXXXXXXXX port 60072 connected to XXXXXXXXXXXX port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 11.3 MBytes 94.5 Mbits/sec
[ 4] 1.00-2.00 sec 10.7 MBytes 90.0 Mbits/sec
[ 4] 2.00-3.00 sec 10.9 MBytes 91.1 Mbits/sec
[ 4] 3.00-4.00 sec 10.8 MBytes 90.2 Mbits/sec
[ 4] 4.00-5.00 sec 10.9 MBytes 91.2 Mbits/sec
[ 4] 5.00-6.00 sec 10.7 MBytes 89.6 Mbits/sec
[ 4] 6.00-7.00 sec 10.9 MBytes 91.2 Mbits/sec
[ 4] 7.00-8.00 sec 10.7 MBytes 90.2 Mbits/sec
[ 4] 8.00-9.00 sec 10.9 MBytes 91.1 Mbits/sec
[ 4] 9.00-10.00 sec 10.7 MBytes 90.0 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 109 MBytes 91.2 Mbits/sec 248 sender
[ 4] 0.00-10.00 sec 109 MBytes 91.0 Mbits/sec receiver

iperf Done.
 
Last edited:
Max

Max

Administrator
Staff member
tonyportersec said:
What actually it is doing? Any side effect?
Click to expand...
By default, one thread is used to distribute stream frames to all the stream subscribers. The tweak we recommended enables multithreaded distribution. This eliminates server issues like feezes and lags when a many subscribers connecting to one stream. Please also read this article describing a typical one-to-many case.
But, if distrubution issues occur, we see low FPS on client side, in WebRTC internals. So, that's not your case according to the following
tonyportersec said:
From the chrome://webrtc-internals,
It is still streaming, but video is freezing.
Click to expand...
We see that server still sending media traffic withhout significant losses.
This seems like local browser issue. Please check CPU load on PC. WebRTC traffic is encrypted, so decryption requires some CPU resources, and may be it is not enough for 17 streams (Edge) + 8 streams (Chrome)
If you plan to play 25 streams per one PC, please consider stream mixing. This requires more server CPU resources, but reduces client channel and CPU load, because client plays only one stream in this case.
 
Max

Max

Administrator
Staff member
We made some tests on Windows 10 PC (i5-8250U 4 physical cores, 20 Mb RAM) playing 25 streams (360p 25 fps, bitrate 1,5-3 Mbps) in 3 Chrome tabs. During the tests CPU load were in range 40-50 % (25% GPU load), without noticeable freezes or stream visual degradation.
So WebRTC playback capacity depends on PC hardware. We do not recommend to play a much streams per one client PC simultaneously. Please consider a mixing many streams to one grid, in this case you can play mixed stream on almost every client and channel. If you need to zoom in one of the streams, you can play it separately in full resolution.
 
T

tonyportersec

Member
Hello,

Can you check, the flashphoner process is now at high CPU usage for a 16 cores PC, but only 1 RTSP is connecting and 2 viewer viewing, and video is very laggy, the RTSP is smooth when direct connect from VLC player.
 
Max

Max

Administrator
Staff member
tonyportersec said:
Can you check, the flashphoner process is now at high CPU usage for a 16 cores PC, but only 1 RTSP is connecting and 2 viewer viewing, and video is very laggy, the RTSP is smooth when direct connect from VLC player.
Click to expand...
Usually, the CPU load may grow due to stream transcoding, or, in the lates JDK 8 and JDK 11 builds, due to network I/O issues. So we recommend to update JDK to 12 or 14 as described here, the following JVM settings tweak is also needed after this update.
If this does not help, please provide SSH access to the server using this form.
 
T

tonyportersec

Member
When high cpu usage and video freezing, In the server log, i got the following error, what does it mean?
CPU is 11th Gen Intel(R) Core(TM) i7-11700 @ 2.50GHz, 16 cores
Memory: 5.65 GiB used / 9.28 GiB cached / 15.28 GiB total
16 rtsp connection without transcoding, and 16 webrtc viewer, all are in resolution 704x576.

server log:

java.lang.reflect.InvocationTargetException
at jdk.internal.reflect.GeneratedMethodAccessor44.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.flashphoner.server.client.handler.MediaHandlerProxyCreator$MediaHandlerInvocationHandler.invoke(Unknown Source)
at com.flashphoner.server.client.handler.wcs4.WCS4Handler_$$_jvst571_0.playAudioRtpPacket(WCS4Handler_$$_jvst571_0.java)
at com.flashphoner.server.remote.E.D.D(Unknown Source)
at com.flashphoner.G.B.messageReceived(Unknown Source)
at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(Unknown Source)
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(Unknown Source)
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(Unknown Source)
at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
at org.jboss.netty.channel.socket.nio.NioWorker.read(Unknown Source)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(Unknown Source)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(Unknown Source)
at org.jboss.netty.channel.socket.nio.NioWorker.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.OutOfMemoryError: Java heap space
05:00:00,244 WARN annelUpstreamHandler - pool-57396-thread-1 EXCEPTION, please implement com.flashphoner.G.B.exceptionCaught() for proper handling.
java.lang.reflect.InvocationTargetException
at jdk.internal.reflect.GeneratedMethodAccessor44.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.flashphoner.server.client.handler.MediaHandlerProxyCreator$MediaHandlerInvocationHandler.invoke(Unknown Source)
at com.flashphoner.server.client.handler.wcs4.WCS4Handler_$$_jvst571_0.playAudioRtpPacket(WCS4Handler_$$_jvst571_0.java)
at com.flashphoner.server.remote.E.D.D(Unknown Source)
at com.flashphoner.G.B.messageReceived(Unknown Source)
at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(Unknown Source)
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(Unknown Source)
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(Unknown Source)
at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
at org.jboss.netty.channel.socket.nio.NioWorker.read(Unknown Source)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(Unknown Source)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(Unknown Source)
at org.jboss.netty.channel.socket.nio.NioWorker.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.OutOfMemoryError: Java heap space
05:00:00,246 INFO RestClient - API-ASYNC-pool-13-thread-1782 SEND REST OBJECT ==>
URL:http://localhost:8081/apps/EchoApp/ConnectionStatusEvent
OBJECT:
{
"nodeId" : "k4c3SWC4KrHhk1RmHb5mKxrQZCGQQNsx@210.17.139.7",
"appKey" : "defaultApp",
"sessionId" : "rtsp://XXXXXXXXXXXXXXXXXXX:554/vid11-97f214f5-e738-4ef2-a09a-d589c4eff373",
"status" : "DISCONNECTED"
}
 
Max

Max

Administrator
Staff member
We checked your report and recommend the following:
1. Update JDK to 12 as described here
2. Enable ZGC as described here
3. Update WCS to build 5.2.969 or newer because 5.2.962 to 5.2.968 may crash while stream decoding (for example, if you play RTSP as VP8).
4. Remove streaming_custom_stream_stress_test configuration from flashphoner.properties
5. Disable NIO network library usage, this should reduce native memory consumption
Code: 
ice_tcp_nio=false[/ICODE]
 
T

tonyportersec

Member
After Updated to 5.2.971
There is a folder created: in the path /usr/local/FlashphonerWebCallServer-5.2.971-217c232d3e203f2e45cb82a28d82ce42a9302df5
it took up all my disk space about 950GB, is it safe to delete it?
 
Max

Max

Administrator
Staff member
tonyportersec said:
There is a folder created: in the path /usr/local/FlashphonerWebCallServer-5.2.971-217c232d3e203f2e45cb82a28d82ce42a9302df5
it took up all my disk space about 950GB, is it safe to delete it?
Click to expand...
This is probable log files. You can stop WCS
Code: 
systemctl stop webcallserver
then clean logs folder and recording folder if there is one
Code: 
rm -rf /usr/local/FlashphonerWebCallServer/logs/*
rm -rf /usr/local/FlashphonerWebCallServer/records/*
 
T

tonyportersec

Member
I have no records, but very big logs file (It took up all my disk space), How to limit the logs size? is it because i am using client_log_level=DEBUG?
 
You must log in or register to reply here.
Top