Streams randomly going black

[AlanM]

Yesterday (3/6/19) we experienced streams randomly going black & no audio. On the V1 server, when we tried to restart the broadcast we got a 'stream already published' error. On the V2 server, we were able to restart the broadcast.

Not all the streams on the server went black. We weren't able to find a cause in the logs.

V1 Server (5.0.2971)
~ 8:30 AM CST - Streams 90258 / 89784
~ 3:45 PM CST - Streams 90126 / 89885 / 90197
~ 5:30 PM CST - Streams 91125

V2 Server (5.1.3762)
~ 3:45 PM CST - Streams 89994
~ 5:30 PM CST - Streams 89994

We emailed in logs / configs

 

[Max]

Hello

This is how you can analyze black streams (version 5.1.3762):
1. Find the black stream using REST API

/rest-api/stream/find
{"name":"89994"}

You should see the stream with PUBLISHING status
2. Check sessionId from found stream:
Example:
"sessionId": "/171.52.46.57:54017/188.40.244.249:8443",
This is unique id of websocket connection.
3. Set this connection to debug.

/logger/enable_client_log
{
"sessionId": "/171.52.46.57:54017/188.40.244.249:8443",
"logLevel": "DEBUG"
}

4. Find debug logs in WCS_HOME/logs/client_logs, zip and send us by email.
5. If you see UDP ports in the debug logs, i.e. 33010
You can capture dump from this UDP port to be sure video traffic is going through.

tcpdump udp port 33010 -i any -s 4096 -w log.pcap

Docs how to capture debug logs from particular connection:
https://docs.flashphoner.com/display/WCS5EN/WCS+Core+logs#WCSCorelogs-Logginglevelmanaging"onthefly"

 

[AlanM]

We experienced the issue on V2 server again today at ~11 AM CST.

Streams 89918 / 91119

Emailed logs in

 

[Max]

Internal ticket WCS-1850 has been created for the issue. Will inform in the thread when there is an update.

 

[Max]

There were errors "Failed to encode compound RTCP packet to send" in your logs.
That issue has been fixed in 5.1.3778. Update to the version is recommended.