Video freeze

T

tonyportersec

Member
The configuration is RTSP -> WCS -> Web broswer.

In my testing, after a few hours of video streaming, some video will freeze until i disconnect and reconnect it again. (I am using chrome latest version, tested on 2 different PC)
If i check it using /rest-api/stream/find_all, it is still connecting.
When the issue happened, i used VLC player to play the RTSP stream without problem.
 
T

tonyportersec

Member
The metrics using rest api shown VIDEO_RATE is 0, i don't know why it will be 0 after run for a few hours. But not all same models of cameras got this problem.

Also, Will WCS reconnect the rtsp automatically if no frame for a given time?


"videoState": {
"muted": false
},

"mediaProvider": "RTSP",
"metrics": {
"VIDEO_SYNC": 1649368996019,
"VIDEO_K_FRAMES": 2149,
"AUDIO_SYNC": 0,
"VIDEO_NACK": 0,
"AUDIO_RATE": 0,
"AUDIO_LOST": 0,
"VIDEO_LOST": 0,
"VIDEO_CODEC": 119,
"VIDEO_B_FRAMES": 0,
"VIDEO_PLI": 0,
"AUDIO_CODEC": 0,
"VIDEO_RATE": 0,
"VIDEO_WIDTH": 704,
"VIDEO_GOP_SIZE": 280,
"VIDEO_HEIGHT": 576,
"VIDEO_FPS": 0,
"VIDEO_P_FRAMES": 632538
}
 
Max

Max

Administrator
Staff member
tonyportersec said:
The metrics using rest api shown VIDEO_RATE is 0, i don't know why it will be 0 after run for a few hours. But not all same models of cameras got this problem.
Click to expand...
Seeme like the camera stops sending media traffic.
tonyportersec said:
Also, Will WCS reconnect the rtsp automatically if no frame for a given time?
Click to expand...
You can enable video RTP activity control
Code: 
rtp_activity_video=true
rtp_activity_timeout=60
In this case, stream shpuld stop if no media traffic form the camera in 60 seconds. Then, it depends on how do you capture RTSP streams.
If you start RTSP capturing by client play request
Code: 
session.createStream({
    name: rtspStreamName,
    display: remoteVideo,
    ....
}).on(STREAM_STATUS.PENDING, function (stream) {
...
}).play();
then you can reconnect by receiving STREAM_STATUS.FAILED event. Please see Streaming Auto Restore example.
If you start RTSP capturing by /rtsp/startup REST API, use REST hook /StreamStatusEvent: if it is received with FAILED state, use REST API to capture the stream again.
 
T

tonyportersec

Member
I tried again, using VLC player to direct connect to the RTSP, and use WCS to connect to the RTSP at the same time, after about a few hrs, the VIDEO RATE is changed to 0, but the VLC player is still playing without problem.
 
Max

Max

Administrator
Staff member
Unfortunately, we cannot reproduce the issue on our test server.
So please collect a debug logs for the problem stream
Code: 
enable_extended_logging=true
client_log_level=debug
then collect a report using report.sh script and send using this form. If report archive is more than 28 M size, please share a link to the archive.
 
T

tonyportersec

Member
The report is sent using your form. Please check. Thanks, there are some rtsp link have video rate is 0, but the rtsp is still connecting using VLC player.
 
Last edited:
Max

Max

Administrator
Staff member
We checked the logs. Seems like WCS starts to receive incomplete video frames from all the RTSP streams at the same moment. So please do the following:
1. Choose one RTSP stream on which the problem is reproducing.
2. Start to collect traffic dump on server
Code: 
tcpdump -npi any -B 10240 -w log.pcap src <RTSP IP address> and src port 554
3. Start to play RTSP stream. There should not be any other RTSP streams on server.
4. When problem is reproduced, stop traffic dump collection
5. Send us the traffic dump, source IP address and RTSP URL using this form. If dump archive is more than 28 M size, please share a link to the archive.
 
T

tonyportersec

Member
I afraid the problem might not reproduce or take very long time to reproduce if only 1 rtsp is connecting at the same time. Anyway, i will try.
 
Max

Max

Administrator
Staff member
Unfortunately, we cannot play the stream from traffic dump because it's filtered by source address, and there are no requesats from WCS to RTSP camera (OPTIONS etc)
Now, we trying to reproduce the issue on our test server with 16 RTSP cameras you've used in your test, and the issue still is not reproduced yet in 3,5 hours.
As we see in logs, all the cameras are on the same IP address (and probably this is the same device). This seems like some network issue between your WCS instance and this device which breaks media packets so they cannot be assembled. Maybe you should change a network or migrate the server closer to the device.
Please check the network if possible using iperf3:
1. Install iperf3 both on WCS and the device
2. On device side, run
Code: 
iperf3 -s -p <port>
The port should be opened
3. On WCS side, run
Code: 
iperf3 -c <device IP> -p <port> -t 60 -R
4. Check the results. If there are any packets losses, this is the reason. Normally, there should not be loss via TCP.
You can also enable incoming RTSP stream buffering
Code: 
rtsp_in_buffer=true
 
T

tonyportersec

Member
The device provides 16 rtsp links is an embedded system, i cannot run iperf3 on it to test.

The device and the WCS are under same NAT, the rtsp link url used in WCS is the public address, and the router port forward all the rtsp connection using 554 to the device under NAT, so what do you suggest for me to test?
 
Max

Max

Administrator
Staff member
tonyportersec said:
The device and the WCS are under same NAT, the rtsp link url used in WCS is the public address, and the router port forward all the rtsp connection using 554 to the device under NAT, so what do you suggest for me to test?
Click to expand...
Please try to exclude router if WCS and the device are under the same NAT. Use private address, not public.
 
Max

Max

Administrator
Staff member
tonyportersec said:
Tired to use private address, the problem still exist. Could you access the server to check the problem?
Click to expand...
We tried to reproduce the issue on our test WCS server (build 5.2.1162), and this is not reproduced during 6,5 hours test.
If you provide SSH access to the server, we try to reproduce it in your environment and collect traffic dump to analyze. But it should not be a production server because we may need to restart a streams capture or WCS itself. If this is possible, please provide SSH access using this form.
 
Max

Max

Administrator
Staff member
Unfortunately, we cannot connect to the server from EU host using credentials you've provided:
Code: 
ssh: connect to host **.***.***.*** port 22: Connection timed out
.
 
You must log in or register to reply here.
Top