Web call server down suddenly

T

tonyportersec

Member
The web call server suddenly down, what should i check? Thanks

- The REST API cannot connect now.
- Cannot start streaming.
- Cannot go to the demo page.

/usr/local/FlashphonerWebCallServer/logs/server_logs$ tail -f /usr/local/FlashphonerWebCallServer/logs/server_logs/flashphoner.log
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(Unknown Source)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(Unknown Source)
at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
at org.jboss.netty.channel.socket.oio_OioWorker.process(Unknown Source)
at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(Unknown Source)
at org.jboss.netty.channel.socket.oio_OioWorker.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)

/usr/local/FlashphonerWebCallServer/logs/server_logs$ pgrep -afn com.flashphoner.server.Server
5759 java -Xmx1024M -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=50999 -Dcom.sun.management.jmxremote.host=localhost -Djava.rmi.server.hostname=localhost -XX:ErrorFile=/usr/local/FlashphonerWebCallServer/logs/error%p.log -Xlog:gc*:/usr/local/FlashphonerWebCallServer/logs/gc-core-2021-09-17_15-16.log:time -XX:+ExplicitGCInvokesConcurrent -Dsun.rmi.dgc.client.gcInterval=36000000000 -Dsun.rmi.dgc.server.gcInterval=36000000000 -Dcom.flashphoner.fms.AppHome=/usr/local/FlashphonerWebCallServer -Djava.library.path=/usr/local/FlashphonerWebCallServer/lib/so:/usr/local/FlashphonerWebCallServer/lib -DWCS_NON_ROOT=true -DsessionDebugEnabled=false -Djdk.tls.client.protocols="TLSv1,TLSv1.1,TLSv1.2" -cp /usr/local/FlashphonerWebCallServer/lib/* com.flashphoner.server.Server
 
Last edited:
Max

Max

Administrator
Staff member
Good day.
Please check if java process hogs over 100% CPU. If so, please check Java version. If you're using Java 8 latest builds (since 242), we recommend to update JDK to 14 as described here.
If the problem persists, please collect a report as described here and send using this form.
 
Max

Max

Administrator
Staff member
In logs we see a possible problem with RTSP sources: seems all the RTSP published sessions have no video:
- server log:
Code: 
05:00:53,189 WARN  RtpActivityTimerTask - FScheduling-pool-1-thread-5 RTP ACTIVITY EVENT DETECTED!
- client logs:
Code: 
05:00:53,434 INFO     PublishSEListener - FScheduling-pool-1-thread-4 onNoRtpActivity RTSP false, no Video RTP activity
Also, seems Java heap memory (1 Gb) is not enough, garbage collector works too often. You have 16 Gb RAM, we recommend to provide Java heap at least 8 Gb. Also we recommend to set up ZGC instead of CMS in wcs-core.properties file:
1. Comment those lines:
Code: 
-Xmx1024M
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=70
2. Add this line
Code: 
-XX:+UnlockExperimentalVMOptions -XX:+UseZGC -Xms8g -Xmx8g
3. Restart WCS.
 
T

tonyportersec

Member
Max said:
In logs we see a possible problem with RTSP sources: seems all the RTSP published sessions have no video:
- server log:
Code: 
05:00:53,189 WARN  RtpActivityTimerTask - FScheduling-pool-1-thread-5 RTP ACTIVITY EVENT DETECTED!
- client logs:
Code: 
05:00:53,434 INFO     PublishSEListener - FScheduling-pool-1-thread-4 onNoRtpActivity RTSP false, no Video RTP activity
Also, seems Java heap memory (1 Gb) is not enough, garbage collector works too often. You have 16 Gb RAM, we recommend to provide Java heap at least 8 Gb. Also we recommend to set up ZGC instead of CMS in wcs-core.properties file:
1. Comment those lines:
Code: 
-Xmx1024M
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=70
2. Add this line
Code: 
-XX:+UnlockExperimentalVMOptions -XX:+UseZGC -Xms8g -Xmx8g
3. Restart WCS.
Click to expand...

But why no video on RTP will cause the service down, it may happen on real situation? And how could we happen this case, e.g) detect it and reconnect the RTSP link?
I have modified the wcs-core.properties as you mentioned.
 
Max

Max

Administrator
Staff member
tonyportersec said:
But why no video on RTP will cause the service down, it may happen on real situation?
Click to expand...
No, this issue is not related to server setup. We mentioned it for your information only. Perhaps you should check those cameras.
tonyportersec said:
And how could we happen this case, e.g) detect it and reconnect the RTSP link?
Click to expand...
You should enable video RTP activity checking if disabled
Code: 
rtp_activity_video=true
In this case, connection to RTSP camera should be closed by RTP activity.
 
T

tonyportersec

Member
Max said:
No, this issue is not related to server setup. We mentioned it for your information only. Perhaps you should check those cameras.

You should enable video RTP activity checking if disabled
Code: 
rtp_activity_video=true
In this case, connection to RTSP camera should be closed by RTP activity.
Click to expand...

For rtp_activity_video, the config file (flashphoner.properties) doesn't contain this line, so it should be true by default. But why the connection is not closed.
 
Max

Max

Administrator
Staff member
tonyportersec said:
For rtp_activity_video, the config file (flashphoner.properties) doesn't contain this line, so it should be true by default. But why the connection is not closed
Click to expand...
Only webrtc connection playing that stream will be closed if no video traffic. But RTSP connection will persist.
So the only way to check if video is stopped is stream metrics REST API request:
Code: 
POST /rest-api/stream/find HTTP/1.1
Host: localhost:8081
Content-Length: 57
Content-Type: application/json

{
    "name":"stream1",
    "published":true,
    "display":["metrics"]
}
If VIDEO_RATE value is 0 during a couple of minutes, this means no video traffic in the stream, so it should be reconnected.
 
T

tonyportersec

Member
Now, i have a problem on streaming video to iPhone and iPad using safari, Chrome on android on my testing and even on demo page, when i login the demo page and go to Stream Diagnostic, and click Start, it shown FAILED, but it was ok before. Do you know why, But it is ok on windows chrome.
 
Last edited:
Max

Max

Administrator
Staff member
tonyportersec said:
Now, i have a problem on streaming video to iPhone and iPad using safari, Chrome on android on my testing and even on demo page, when i login the demo page and go to Stream Diagnostic, and click Start, it shown FAILED, but it was ok before. Do you know why, But it is ok on windows chrome.
Click to expand...
Please reproduce the problem, collect a report as described here and send using this form.
 
Max

Max

Administrator
Staff member
We received you report. In the logs we see connections only from Chrome 94 on Windows, and all the connections with FAILED status are finished by client session disconnection:
Code: 
2021-10-25 08:36:48;WebRTC;gns01577_1;a0ec0150-352b-11ec-8a63-cd4b4abcbf95;00:19:31;FAILED;Stopped by session disconnect;;SUBSCRIBE;0;/210.17.139.78:58776/210.17.139.7:8443-9a425c21-5b7c-478c-9c9a-5ce2753953e7;
2021-10-25 08:36:47;WebRTC;gns01577_2;9fdb0680-352b-11ec-8a63-cd4b4abcbf95;00:19:33;FAILED;Stopped by session disconnect;;SUBSCRIBE;0;/210.17.139.78:55597/210.17.139.7:8443-2fb639c3-83e2-422d-9efc-7c52e5c251ec;
It seems like network problem on client side. But you mentioned iOS and Android clients, there are no such records in logs.
So you should:
- open demo page in iOS Safari or Android Chrome (Two Way Streaming is preferrable)
- reproduce the issue (publish or play a stream with FAILED message)
- collect a report and send.
Two Way Streaming example should also display an extended info about FAILED status, this may help to diagnose.
Or you can provide SSH access to the server using this form, in this case, we will try to test.
 
T

tonyportersec

Member
just tested the two-way streaming on my iPad, iPhone and tried wifi and 4G network, when click Connect button, it shown FAILED, the report is sent.
 
Max

Max

Administrator
Staff member
tonyportersec said:
just tested the two-way streaming on my iPad, iPhone and tried wifi and 4G network, when click Connect button, it shown FAILED, the report is sent.
Click to expand...
There are no connections from iOS in logs provided:
Code: 
  ...
  "clientOSVersion" : "5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36",
  "clientBrowserVersion" : "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36",
  ...
Seems this is a browser side issue. Check browser console for errors. Also please clarify what iOS version do you test.
You can also try to test on our demo server in Two Way Streaming example https://demo.flashphoner.com:8888/c...ming/two_way_streaming/two_way_streaming.html
If the issue is reproducing, this is definitely device issue. Try to cahnge network or change device. If demo example on our server works, but not on your server, we need access to it to find a possible reason. Please provide access to the server using this form.
 
You must log in or register to reply here.
Top