Web call server down suddenly

The web call server suddenly down, what should i check? Thanks

- The REST API cannot connect now.
- Cannot start streaming.
- Cannot go to the demo page.

/usr/local/FlashphonerWebCallServer/logs/server_logs$ tail -f /usr/local/FlashphonerWebCallServer/logs/server_logs/flashphoner.log
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(Unknown Source)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(Unknown Source)
at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
at org.jboss.netty.channel.socket.oio_OioWorker.process(Unknown Source)
at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(Unknown Source)
at org.jboss.netty.channel.socket.oio_OioWorker.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)

/usr/local/FlashphonerWebCallServer/logs/server_logs$ pgrep -afn com.flashphoner.server.Server
5759 java -Xmx1024M -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=50999 -Dcom.sun.management.jmxremote.host=localhost -Djava.rmi.server.hostname=localhost -XX:ErrorFile=/usr/local/FlashphonerWebCallServer/logs/error%p.log -Xlog:gc*:/usr/local/FlashphonerWebCallServer/logs/gc-core-2021-09-17_15-16.log:time -XX:+ExplicitGCInvokesConcurrent -Dsun.rmi.dgc.client.gcInterval=36000000000 -Dsun.rmi.dgc.server.gcInterval=36000000000 -Dcom.flashphoner.fms.AppHome=/usr/local/FlashphonerWebCallServer -Djava.library.path=/usr/local/FlashphonerWebCallServer/lib/so:/usr/local/FlashphonerWebCallServer/lib -DWCS_NON_ROOT=true -DsessionDebugEnabled=false -Djdk.tls.client.protocols="TLSv1,TLSv1.1,TLSv1.2" -cp /usr/local/FlashphonerWebCallServer/lib/* com.flashphoner.server.Server
 
Last edited:

Max

Administrator
Staff member
Good day.
Please check if java process hogs over 100% CPU. If so, please check Java version. If you're using Java 8 latest builds (since 242), we recommend to update JDK to 14 as described here.
If the problem persists, please collect a report as described here and send using this form.
 

Max

Administrator
Staff member
In logs we see a possible problem with RTSP sources: seems all the RTSP published sessions have no video:
- server log:
Code:
05:00:53,189 WARN  RtpActivityTimerTask - FScheduling-pool-1-thread-5 RTP ACTIVITY EVENT DETECTED!
- client logs:
Code:
05:00:53,434 INFO     PublishSEListener - FScheduling-pool-1-thread-4 onNoRtpActivity RTSP false, no Video RTP activity
Also, seems Java heap memory (1 Gb) is not enough, garbage collector works too often. You have 16 Gb RAM, we recommend to provide Java heap at least 8 Gb. Also we recommend to set up ZGC instead of CMS in wcs-core.properties file:
1. Comment those lines:
Code:
-Xmx1024M
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=70
2. Add this line
Code:
-XX:+UnlockExperimentalVMOptions -XX:+UseZGC -Xms8g -Xmx8g
3. Restart WCS.
 
In logs we see a possible problem with RTSP sources: seems all the RTSP published sessions have no video:
- server log:
Code:
05:00:53,189 WARN  RtpActivityTimerTask - FScheduling-pool-1-thread-5 RTP ACTIVITY EVENT DETECTED!
- client logs:
Code:
05:00:53,434 INFO     PublishSEListener - FScheduling-pool-1-thread-4 onNoRtpActivity RTSP false, no Video RTP activity
Also, seems Java heap memory (1 Gb) is not enough, garbage collector works too often. You have 16 Gb RAM, we recommend to provide Java heap at least 8 Gb. Also we recommend to set up ZGC instead of CMS in wcs-core.properties file:
1. Comment those lines:
Code:
-Xmx1024M
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=70
2. Add this line
Code:
-XX:+UnlockExperimentalVMOptions -XX:+UseZGC -Xms8g -Xmx8g
3. Restart WCS.

But why no video on RTP will cause the service down, it may happen on real situation? And how could we happen this case, e.g) detect it and reconnect the RTSP link?
I have modified the wcs-core.properties as you mentioned.
 

Max

Administrator
Staff member
But why no video on RTP will cause the service down, it may happen on real situation?
No, this issue is not related to server setup. We mentioned it for your information only. Perhaps you should check those cameras.
And how could we happen this case, e.g) detect it and reconnect the RTSP link?
You should enable video RTP activity checking if disabled
Code:
rtp_activity_video=true
In this case, connection to RTSP camera should be closed by RTP activity.
 
No, this issue is not related to server setup. We mentioned it for your information only. Perhaps you should check those cameras.

You should enable video RTP activity checking if disabled
Code:
rtp_activity_video=true
In this case, connection to RTSP camera should be closed by RTP activity.

For rtp_activity_video, the config file (flashphoner.properties) doesn't contain this line, so it should be true by default. But why the connection is not closed.
 

Max

Administrator
Staff member
For rtp_activity_video, the config file (flashphoner.properties) doesn't contain this line, so it should be true by default. But why the connection is not closed
Only webrtc connection playing that stream will be closed if no video traffic. But RTSP connection will persist.
So the only way to check if video is stopped is stream metrics REST API request:
Code:
POST /rest-api/stream/find HTTP/1.1
Host: localhost:8081
Content-Length: 57
Content-Type: application/json

{
    "name":"stream1",
    "published":true,
    "display":["metrics"]
}
If VIDEO_RATE value is 0 during a couple of minutes, this means no video traffic in the stream, so it should be reconnected.
 
Now, i have a problem on streaming video to iPhone and iPad using safari, Chrome on android on my testing and even on demo page, when i login the demo page and go to Stream Diagnostic, and click Start, it shown FAILED, but it was ok before. Do you know why, But it is ok on windows chrome.
 
Last edited:

Max

Administrator
Staff member
Now, i have a problem on streaming video to iPhone and iPad using safari, Chrome on android on my testing and even on demo page, when i login the demo page and go to Stream Diagnostic, and click Start, it shown FAILED, but it was ok before. Do you know why, But it is ok on windows chrome.
Please reproduce the problem, collect a report as described here and send using this form.
 

Max

Administrator
Staff member
We received you report. In the logs we see connections only from Chrome 94 on Windows, and all the connections with FAILED status are finished by client session disconnection:
Code:
2021-10-25 08:36:48;WebRTC;gns01577_1;a0ec0150-352b-11ec-8a63-cd4b4abcbf95;00:19:31;FAILED;Stopped by session disconnect;;SUBSCRIBE;0;/210.17.139.78:58776/210.17.139.7:8443-9a425c21-5b7c-478c-9c9a-5ce2753953e7;
2021-10-25 08:36:47;WebRTC;gns01577_2;9fdb0680-352b-11ec-8a63-cd4b4abcbf95;00:19:33;FAILED;Stopped by session disconnect;;SUBSCRIBE;0;/210.17.139.78:55597/210.17.139.7:8443-2fb639c3-83e2-422d-9efc-7c52e5c251ec;
It seems like network problem on client side. But you mentioned iOS and Android clients, there are no such records in logs.
So you should:
- open demo page in iOS Safari or Android Chrome (Two Way Streaming is preferrable)
- reproduce the issue (publish or play a stream with FAILED message)
- collect a report and send.
Two Way Streaming example should also display an extended info about FAILED status, this may help to diagnose.
Or you can provide SSH access to the server using this form, in this case, we will try to test.
 
just tested the two-way streaming on my iPad, iPhone and tried wifi and 4G network, when click Connect button, it shown FAILED, the report is sent.
 

Max

Administrator
Staff member
just tested the two-way streaming on my iPad, iPhone and tried wifi and 4G network, when click Connect button, it shown FAILED, the report is sent.
There are no connections from iOS in logs provided:
Code:
  ...
  "clientOSVersion" : "5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36",
  "clientBrowserVersion" : "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36",
  ...
Seems this is a browser side issue. Check browser console for errors. Also please clarify what iOS version do you test.
You can also try to test on our demo server in Two Way Streaming example https://demo.flashphoner.com:8888/c...ming/two_way_streaming/two_way_streaming.html
If the issue is reproducing, this is definitely device issue. Try to cahnge network or change device. If demo example on our server works, but not on your server, we need access to it to find a possible reason. Please provide access to the server using this form.
 
Top