Server Crash

[RafaelRibeiro]

Hello!

This night we got a server crash and looking at logs we found only this exception from java:

00:35:20,909 ERROR TcpClient - TcpClient-/X.X.X.X:31042-pool-76-thread-1 Got exception
java.lang.NullPointerException: message
at org.jboss.netty.channel.UpstreamMessageEvent.<init>(Unknown Source)
at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
at com.flashphoner.ice.A.G.handleUpstream(Unknown Source)
at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(Unknown Source)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(Unknown Source)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(Unknown Source)
at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
at org.jboss.netty.channel.socket.nio.NioWorker.read(Unknown Source)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(Unknown Source)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(Unknown Source)
at org.jboss.netty.channel.socket.nio.NioWorker.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

 

[RafaelRibeiro]

I sent the logs by upload link.
Thanks

 

[Max]

Good day.
First, please use "Report" button on your topic page to send a report. Do not use report links from another topics.
About server crash, please clarify the following:
- WCS build
- JDK version used
- Is custom mixer layout used? If yes, please provide the source code
Also, please provide the following:
- crash log file /usr/local/FlashphonerWebCallServer/logs/error*.log
- server current settings
Send all required files using this link

 

[RafaelRibeiro]

Hello!

I have sent a wetransfer link with logs and settings file to be download.

WCS Build: 5.2.665
JDK:

openjdk version "1.8.0_252"
OpenJDK Runtime Environment (build 1.8.0_252-b09)
OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)

- Is custom mixer layout used? If yes, please provide the source code
Yes, sent the file too.

Settings and logs already sent too.

Scenario:

Mixer crash or shutdown.

We publish on mixer 1 RTMP stream as Desktop and 2 webcams.
After 40 minuts, mixer starts to lost audio and video sync and then fail - stops.

We can not identify exactly whats happen, we just know mixer goes down and we needs to startup the mixer again.

 

[RafaelRibeiro]

We found the possible root cause of this problem.

Our recording script send files over RSYNC each 300 MB.
Once 300 MB is reached and starts to send the video record from mixer, the mixer frozes and after sent the file, back with non sync audio and video.

STREAM_NAME=$1
SRC_FILE=$2
SRC_DIR="/usr/local/FlashphonerWebCallServer/records/"
REPLACE_STR="/mnt/videos/$STREAM_NAME-"
DST_FILE="${SRC_FILE/$SRC_DIR/$REPLACE_STR}"
#cp $SRC_FILE $DST_FILE
rsync -avz -e "ssh -i /root/user_key -p 2222" $SRC_FILE user@remotehost:$DST_FILE


Disable rsync and mixer works normally.

 

[Max]

Our recording script send files over RSYNC each 300 MB.
Once 300 MB is reached and starts to send the video record from mixer, the mixer frozes and after sent the file, back with non sync audio and video.

Seems like the rsync operation consumes most of channel bandwidth, so freezes andframe losses occur in streams published. According to mixer settings, you're using non-realtime mixer

mixer_realtime=false
...
mixer_out_buffer_enabled=true
mixer_out_buffer_initial_size=3500
mixer_video_buffer_length=2000
mixer_out_buffer_polling_time=800
mixer_out_buffer_start_size=1500

so if at least one of incoming streams freezes then mixer outgoing stream will freeze too. In its turn, after freeze audio will be out of sync with video until the a/v sync is aligned again.
We recommend the following:
1. Enable realtime mixer

mixer_realtime=true

In this case, incoming streams can loose quality, but mixer outgoing stream not (however, if all incoming stream freeze, it will lok as mixer freeze)
2. Expand server channel bandwidth. For example, if this is 1 Gbps, expand it to 10 Gbps
3. Move rsync target server to the same datacenter where WCS is running.
4. Use lower resolution/bitrate for publishing streams and mixer output stream. Now publishsing bitrate looks too expensive (if you do not publishing VR streams)

webrtc_sdp_max_bitrate_bps=3000000
...
mixer_video_bitrate_kbps=2000

5. If the above does not help, exclude rsync from workflow