Flashphoner hang frequently

[Stanley]

Hi Max,

I encounter 2 x load balanced Flashphoner WCS v5.1.3253 hang almost every 2 days. I have to restart daily. These server ingest 18 RTMP streams (640x360 resolution 200kbps bitrate) and have about 80 WSPlayer concurrent player on each server. Whenever it hang we're unable to access to admin page as per attached screenshot. All user unable to play video until we restart webcallserver service.

The flashphoner is installed in ubuntu 18.04 LTS with server specs below.
Dell PowerEdge R420
2 x Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
32GB RAM
2 x 300GB SAS 15KRPM (RAID1)

I also noticed that our server is always 20 to 30% CPU usage without any players.
Please let me know what logs do you require to troubleshoot.

Thanks.

 

[Max]

Hello
This is line from flashphoner_manager.log

03:05:49,763 ERROR agerRemoteRmiService - RMI TCP Connection(30)-{ip} Invalid response on connect method, response should contain Connection object. Using object received from client

You are using REST hooks.
1. Make sure your REST hooks work properly and return valid connection object always
2. If you see hanging again, try to test your REST url using curl or REST console https://chrome.google.com/webstore/detail/advanced-rest-client/hgmloofddffdnphfgcellkdfbfbjeloo. Make sure it works on that moment.
2. Try to disable REST hooks and set default app url

update app defaultApp -l http://localhost:9091/EchoApp

According the docs, REST hook /connect must always return a valid result.
If returned result is incomplete or failed, connection will not be established.
https://docs.flashphoner.com/display/WCS5EN/Type 1 - the connect method

 

[Max]

For a deep troubleshooting you can gather logs as described here (without tcpdump):
https://docs.flashphoner.com/display/WCS5EN/Preparing+an+error+report
We need ll logs covering at least 1 hour before the hanging.

 

[Stanley]

Hi Max,
Thanks for your reply. When it hangs I cant even access to admin page error 500 as per screenshot. I also cant telnet to port 8081 and access to http://ServerIP:8081/?action=stat

Please advise on this. I believe its nothing to do with Rest hook as it is working from the beginning of server startup until 2 days it hangs.

 

[Max]

1. We need all logs in the folder WCS_HOME/logs and child folders
Please zip and share all logs. If archive is too big, try to include last 12 hours logs only.
2. We need folder WCS_HOME/conf
3. If you encounter with the hanging again, please do jstack report for core process:
https://docs.flashphoner.com/display/WCS5EN/Accessory tools#Accessorytools-jstacktool
You can send all reports to support@flashphoner.com

 

[Stanley]

Hi Max, I've sent you 2 of our server logs hang at the same time 2 days ago. Do you have any feedback?

 

[Max]

Hello
We have not found any errors in the logs.
It looks like the core process was killed by the system.
Please check /var/log/messages and physical memory usage (RSS).
You can also send SSH access to support@flashphoner.com. We will check.

 

[Stanley]

Hi Max, We also encounter WCS unable to auto startup installed on ubuntu 18.04 LTS. Have to manually start.
Installed version WCS v5.1.3235 and also tested v5.1.3375

Can you test on your environment ?

 

[Max]

WCS unable to auto startup installed on ubuntu 18.04 LTS

We have created internal ticket WCS-1335. We will report once we have any progress here.

 

[Max]

Hello
It is fixed in 5.1.3356

 

[Stanley]

Hi Max,
Do you publish release note on each version?

 

[Max]

We are preparing change log.
Once it is done, we will share the link.