Stream is getting failed/disconnected

[jyotipatidar]

Hi Max,
suddenly our stream started getting disconnect after every minute or two, please suggest it's a prod issue
13:13:12,806 INFO WSServerHandler - WS-pool-19-thread-214 Orgign: null
13:13:12,807 WARN WSServerHandler - WS-pool-19-thread-214 Close channel [id: 0x74bb5297, /<ip>:39994 => /<ip>:8080] because: org.jboss.netty.handler.codec.http.websocketx.WebSocketHandshakeException 'not a WebSocket handshake request: missing upgrade'
13:24:06,119 ERROR HttpServerHandler - HTTP-pool-2-thread-728 HTTP error
java.lang.IllegalArgumentException: empty text
at org.jboss.netty.handler.codec.http.HttpVersion.<init>(Unknown Source)
at org.jboss.netty.handler.codec.http.HttpVersion.valueOf(Unknown Source)
at org.jboss.netty.handler.codec.http.HttpRequestDecoder.createMessage(Unknown Source)
at org.jboss.netty.handler.codec.http.HttpMessageDecoder.decode(Unknown Source)
at org.jboss.netty.handler.codec.http.HttpMessageDecoder.decode(Unknown Source)
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(Unknown Source)
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(Unknown Source)
at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
at org.jboss.netty.channel.socket.nio.NioWorker.read(Unknown Source)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(Unknown Source)
at org.jboss.netty.channel.socket.nio.DeadlockAwareNioWorker.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:835)
13:24:06,119 ERROR HttpServerHandler - HTTP-pool-2-thread-728 HTTP error
java.lang.IllegalArgumentException: empty text

 

[Max]

Hello,

Please collect a report as described here and send us using this link, or provide us SSH access to your server using this link.

 

[jyotipatidar]

sent

 

[Max]

1. Please collect DEBUG logs

client_log_level=DEBUG

in flashphoner.properties

and tcpdump

tcpdump -i any -s 0 -B 10240 -w log.pcap

Preparing an error report - Web Call Server 5.2 - EN - Flashphoner Documentation

docs.flashphoner.com

2. Please submit full report.

cd /usr/local/FlashphonerWebCallServer/tools
sudo ./report.sh --sysinfo --conf --tar

For now we can see INFO logs and no report attached.

 

[jyotipatidar]

Hi Max,
Today we are not able to reproduce this issue yet, I have enabled the debug log. Once we face issue I will share the report.
But at the first look it seems to be internet slowness issue, can we do something like this ?
When session is running and internet speed becomes slow store the recording in browser cache and then send it when speed is back ? Because all the time internet speed can not be constant. We are recording educational videos and due to this disconnect all video are getting generated in parts do we have any alternate for this issue so that 1 hour or 30 min recording is stored in 1 singe video instead of parts ?

 

[Max]

But at the first look it seems to be internet slowness issue, can we do something like this ?

When publisher channel quality issues occur, there is the only workaround to use lower stream resolution/bitrate. You can try switch to TCP transport, but, if this does not help, you have to choose lower resolutions.
For example, if you're publishing 720p stream, try to publish 480p or 360p with bitrate no more than 1000 kbps:

    session.createStream({
        name: streamName,
        display: localVideo,
        receiveVideo: false,
        receiveAudio: false,
        constraints: {
               video: {
                      width: 640,
                      height: 360,
                      minBitrate: 100,
                      maxBitrate: 1000
               }
        }
        ...
   }).publish()

Another option (for example if you're publishing screen share streams) is migrate the server to datacenter which is closest to publisher.
Also please do not use mobile internet (especially 3G anl lower) for publishing.

 

[RafaelRibeiro]

How to enable TCP transport and whats beneficts?

 

[jyotipatidar]

Max has already shared the link in his reply it has Configuration section

Publishing and playing stream via WebRTC over TCP - Web Call Server 5.2 - EN - Flashphoner Documentation

docs.flashphoner.com

@Max
We already use TCP transport, and bit rate if we decrease it decreases the shared screen quality, it look blur

1) also all of sudden Server EBS 15 GB is full, not sure what is using this much space, though we deleted all recorded video but still i see 15 GB used.
Do we have any tmp or something which is using space ?

 

[RafaelRibeiro]

@jyotipatidar, thanks!

 

[Max]

We already use TCP transport, and bit rate if we decrease it decreases the shared screen quality, it look blur

In this case, you have to migrate server instance maximally closer to publisher, check the publishers last mile to prevent any channel quality issues.

1) also all of sudden Server EBS 15 GB is full, not sure what is using this much space, though we deleted all recorded video but still i see 15 GB used.
Do we have any tmp or something which is using space ?

Please disable client logs when they are not needed:

enable_extended_logging=false

Then stop the server, clean logs and start it again

sudo service webcallserver stop
sudo rm -rf /usr/local/FlashphonerWebCallServer/logs/*
sudo service webcallserver start

 

[Max]

How to enable TCP transport and whats beneficts?

The benefit of TCP transport is packet losses prevention. But the latency can slightly grow up (1-3 seconds), this is drawback.

 

[jyotipatidar]

Thanks Max, yea the issue was due to
client_log_level=DEBUG
it was consuming lot of space in the logs, I deleted all the logs and disabled Debug log level.

Thanks again !