Web call server down suddenly

The web call server suddenly down, what should i check? Thanks

- The REST API cannot connect now.
- Cannot start streaming.
- Cannot go to the demo page.

/usr/local/FlashphonerWebCallServer/logs/server_logs$ tail -f /usr/local/FlashphonerWebCallServer/logs/server_logs/flashphoner.log
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(Unknown Source)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(Unknown Source)
at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
at org.jboss.netty.channel.socket.oio_OioWorker.process(Unknown Source)
at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(Unknown Source)
at org.jboss.netty.channel.socket.oio_OioWorker.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)

/usr/local/FlashphonerWebCallServer/logs/server_logs$ pgrep -afn com.flashphoner.server.Server
5759 java -Xmx1024M -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=50999 -Dcom.sun.management.jmxremote.host=localhost -Djava.rmi.server.hostname=localhost -XX:ErrorFile=/usr/local/FlashphonerWebCallServer/logs/error%p.log -Xlog:gc*:/usr/local/FlashphonerWebCallServer/logs/gc-core-2021-09-17_15-16.log:time -XX:+ExplicitGCInvokesConcurrent -Dsun.rmi.dgc.client.gcInterval=36000000000 -Dsun.rmi.dgc.server.gcInterval=36000000000 -Dcom.flashphoner.fms.AppHome=/usr/local/FlashphonerWebCallServer -Djava.library.path=/usr/local/FlashphonerWebCallServer/lib/so:/usr/local/FlashphonerWebCallServer/lib -DWCS_NON_ROOT=true -DsessionDebugEnabled=false -Djdk.tls.client.protocols="TLSv1,TLSv1.1,TLSv1.2" -cp /usr/local/FlashphonerWebCallServer/lib/* com.flashphoner.server.Server
 
Last edited:

Max

Administrator
Staff member
Good day.
Please check if java process hogs over 100% CPU. If so, please check Java version. If you're using Java 8 latest builds (since 242), we recommend to update JDK to 14 as described here.
If the problem persists, please collect a report as described here and send using this form.
 

Max

Administrator
Staff member
In logs we see a possible problem with RTSP sources: seems all the RTSP published sessions have no video:
- server log:
Code:
05:00:53,189 WARN  RtpActivityTimerTask - FScheduling-pool-1-thread-5 RTP ACTIVITY EVENT DETECTED!
- client logs:
Code:
05:00:53,434 INFO     PublishSEListener - FScheduling-pool-1-thread-4 onNoRtpActivity RTSP false, no Video RTP activity
Also, seems Java heap memory (1 Gb) is not enough, garbage collector works too often. You have 16 Gb RAM, we recommend to provide Java heap at least 8 Gb. Also we recommend to set up ZGC instead of CMS in wcs-core.properties file:
1. Comment those lines:
Code:
-Xmx1024M
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=70
2. Add this line
Code:
-XX:+UnlockExperimentalVMOptions -XX:+UseZGC -Xms8g -Xmx8g
3. Restart WCS.
 
In logs we see a possible problem with RTSP sources: seems all the RTSP published sessions have no video:
- server log:
Code:
05:00:53,189 WARN  RtpActivityTimerTask - FScheduling-pool-1-thread-5 RTP ACTIVITY EVENT DETECTED!
- client logs:
Code:
05:00:53,434 INFO     PublishSEListener - FScheduling-pool-1-thread-4 onNoRtpActivity RTSP false, no Video RTP activity
Also, seems Java heap memory (1 Gb) is not enough, garbage collector works too often. You have 16 Gb RAM, we recommend to provide Java heap at least 8 Gb. Also we recommend to set up ZGC instead of CMS in wcs-core.properties file:
1. Comment those lines:
Code:
-Xmx1024M
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=70
2. Add this line
Code:
-XX:+UnlockExperimentalVMOptions -XX:+UseZGC -Xms8g -Xmx8g
3. Restart WCS.

But why no video on RTP will cause the service down, it may happen on real situation? And how could we happen this case, e.g) detect it and reconnect the RTSP link?
I have modified the wcs-core.properties as you mentioned.
 

Max

Administrator
Staff member
But why no video on RTP will cause the service down, it may happen on real situation?
No, this issue is not related to server setup. We mentioned it for your information only. Perhaps you should check those cameras.
And how could we happen this case, e.g) detect it and reconnect the RTSP link?
You should enable video RTP activity checking if disabled
Code:
rtp_activity_video=true
In this case, connection to RTSP camera should be closed by RTP activity.
 
No, this issue is not related to server setup. We mentioned it for your information only. Perhaps you should check those cameras.

You should enable video RTP activity checking if disabled
Code:
rtp_activity_video=true
In this case, connection to RTSP camera should be closed by RTP activity.

For rtp_activity_video, the config file (flashphoner.properties) doesn't contain this line, so it should be true by default. But why the connection is not closed.
 

Max

Administrator
Staff member
For rtp_activity_video, the config file (flashphoner.properties) doesn't contain this line, so it should be true by default. But why the connection is not closed
Only webrtc connection playing that stream will be closed if no video traffic. But RTSP connection will persist.
So the only way to check if video is stopped is stream metrics REST API request:
Code:
POST /rest-api/stream/find HTTP/1.1
Host: localhost:8081
Content-Length: 57
Content-Type: application/json

{
    "name":"stream1",
    "published":true,
    "display":["metrics"]
}
If VIDEO_RATE value is 0 during a couple of minutes, this means no video traffic in the stream, so it should be reconnected.
 
Top