WCS service issue with AWS Markekplace AMI

manikandan · Jul 14, 2021

Hello,

I launched WCS server using AWS marketplace AMI https://aws.amazon.com/marketplace/reviews/reviews-list/B07ZVRPH5G and done similar to this setup.

Also updated the WCS package to latest version using the bash script in user data.

Upon launching the instance, when I checked with the webcallserver status I noticed some errors in the service

Code:

[ec2-user@ip-172-31-70-30 ~]$ sudo service webcallserver status -l
Redirecting to /bin/systemctl status  -l webcallserver.service
● webcallserver.service - Flashphoner WebCallServer
   Loaded: loaded (/etc/systemd/system/webcallserver.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2021-07-14 16:27:59 UTC; 14s ago
  Process: 4306 ExecStop=/bin/bash webcallserver stop (code=exited, status=0/SUCCESS)
 Main PID: 4995 (java)
   CGroup: /system.slice/webcallserver.service
           ‣ 4995 java -Xmx512M -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=50999 -Dcom.sun.management.jmxremote.host=localhost -Djava.rmi.server.hostname=localhost -XX:ErrorFile=/usr/local/FlashphonerWebCallServer/logs/error%p.log -Xlog:gc*:/usr/local/FlashphonerWebCallServer/logs/gc-core-2021-07-14_16-27.log:time -XX:+ExplicitGCInvokesConcurrent -Dsun.rmi.dgc.client.gcInterval=36000000000 -Dsun.rmi.dgc.server.gcInterval=36000000000 -Dcom.flashphoner.fms.AppHome=/usr/local/FlashphonerWebCallServer -Djava.library.path=/usr/local/FlashphonerWebCallServer/lib/so:/usr/local/FlashphonerWebCallServer/lib -DWCS_NON_ROOT=true -DsessionDebugEnabled=false -Djdk.tls.client.protocols="TLSv1,TLSv1.1,TLSv1.2" -cp /usr/local/FlashphonerWebCallServer/lib/* com.flashphoner.server.Server

Jul 14 16:28:00 ip-172-31-70-30.ec2.internal sudo[4927]:     root : TTY=unknown ; PWD=/usr/local/FlashphonerWebCallServer-5.2.983-39b59ee9dba7df2a6163fdc2eb92aeb72bb90ebb/bin ; USER=flashphoner ; COMMAND=/bin/env LD_LIBRARY_PATH=/usr/local/FlashphonerWebCallServer/lib/so java -Xmx512M -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=50999 -Dcom.sun.management.jmxremote.host=localhost -Djava.rmi.server.hostname=localhost -XX:ErrorFile=/usr/local/FlashphonerWebCallServer/logs/error%p.log -Xlog:gc*:/usr/local/FlashphonerWebCallServer/logs/gc-core-2021-07-14_16-27.log:time -XX:+ExplicitGCInvokesConcurrent -Dsun.rmi.dgc.client.gcInterval=36000000000 -Dsun.rmi.dgc.server.gcInterval=36000000000 -Dcom.flashphoner.fms.AppHome=/usr/local/FlashphonerWebCallServer -cp /usr/local/FlashphonerWebCallServer/lib/*
Jul 14 16:28:00 ip-172-31-70-30.ec2.internal sudo[4927]:     root : (command continued) com.flashphoner.server.license.ExpiredUpdates
Jul 14 16:28:02 ip-172-31-70-30.ec2.internal sudo[4979]:     root : TTY=unknown ; PWD=/usr/local/FlashphonerWebCallServer-5.2.983-39b59ee9dba7df2a6163fdc2eb92aeb72bb90ebb/bin ; USER=flashphoner ; COMMAND=/bin/echo -e [2021-07-14 16:28:02] INFO startWithSudo - Starting FlashphonerWebCallServer as user flashphoner
Jul 14 16:28:02 ip-172-31-70-30.ec2.internal sudo[4981]:     root : TTY=unknown ; PWD=/usr/local/FlashphonerWebCallServer-5.2.983-39b59ee9dba7df2a6163fdc2eb92aeb72bb90ebb/bin ; USER=flashphoner ; COMMAND=/bin/env LD_LIBRARY_PATH=/usr/local/FlashphonerWebCallServer/lib/so java -Xmx512M -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=50999 -Dcom.sun.management.jmxremote.host=localhost -Djava.rmi.server.hostname=localhost -XX:ErrorFile=/usr/local/FlashphonerWebCallServer/logs/error%p.log -Xlog:gc*:/usr/local/FlashphonerWebCallServer/logs/gc-core-2021-07-14_16-27.log:time -XX:+ExplicitGCInvokesConcurrent -Dsun.rmi.dgc.client.gcInterval=36000000000 -Dsun.rmi.dgc.server.gcInterval=36000000000 -Dcom.flashphoner.fms.AppHome=/usr/local/FlashphonerWebCallServer
Jul 14 16:28:02 ip-172-31-70-30.ec2.internal sudo[4981]:     root : (command continued) -Djava.library.path=/usr/local/FlashphonerWebCallServer/lib/so:/usr/local/FlashphonerWebCallServer/lib -DWCS_NON_ROOT=true -DsessionDebugEnabled=false -Djdk.tls.client.protocols="TLSv1,TLSv1.1,TLSv1.2" -cp /usr/local/FlashphonerWebCallServer/lib/* com.flashphoner.server.Server
Jul 14 16:28:04 ip-172-31-70-30.ec2.internal sudo[5057]:     root : TTY=unknown ; PWD=/usr/local/FlashphonerWebCallServer-5.2.983-39b59ee9dba7df2a6163fdc2eb92aeb72bb90ebb/bin ; USER=flashphoner ; COMMAND=/bin/touch /usr/local/FlashphonerWebCallServer/bin/FlashphonerWebCallServer
Jul 14 16:28:10 ip-172-31-70-30.ec2.internal sudo[5143]:     root : TTY=unknown ; PWD=/usr/local/FlashphonerWebCallServer-5.2.983-39b59ee9dba7df2a6163fdc2eb92aeb72bb90ebb/bin ; USER=flashphoner ; COMMAND=/bin/echo -e [2021-07-14 16:28:10] ERROR start - FlashphonerWebCallServer started, but is not healthy, please try to restart
Jul 14 16:28:10 ip-172-31-70-30.ec2.internal bash[4401]: FlashphonerWebCallServer: startingFlashphonerWebCallServer started, but is not healthy, please try to restart
Jul 14 16:28:10 ip-172-31-70-30.ec2.internal bash[4401]: [FAILED]
Jul 14 16:28:10 ip-172-31-70-30.ec2.internal systemd[1]: webcallserver.service: Supervising process 4995 which is not our child. We'll most likely not notice when it exits.

it seems some kind of permission issue when running service start command as root.

Let me know how it can be fixed?

Max · Jul 15, 2021

Good day.
Please try the following

Code:

sudo /usr/local/FlashphonerWebCallServer/webcallserver set-permissions
sudo systemctl restart webcallserver

manikandan · Jul 15, 2021

Max said:
sudo /usr/local/FlashphonerWebCallServer/webcallserver set-permissions

I think its sudo /usr/local/FlashphonerWebCallServer/bin/webcallserver set-permissions

Even after setting the permissions, still it has same error as above mentioned in the ticket. I have attached the script file

I am planning to use Marketplace AMI setup. Hoping to get this fixed.

Max · Jul 15, 2021

We cannot reproduce the issue starting a new instance from latest Marketplace AMI (5.2.944-systemd246) with your userdata script: WCS is started successfully without any permission fixes.
Please provide SSH acceess to the problem instance using this form, we will check

manikandan · Jul 15, 2021

Max said:
Please provide SSH acceess to the problem instance using this form, we will check

Can you please share your IP address so that I can restrict SSH access?

Max · Jul 15, 2021

manikandan said:
Can you please share your IP address so that I can restrict SSH access?

Please see at this page (hidden by spoiler).

manikandan · Jul 15, 2021

Shared the ssh access. Please check.

Max · Jul 15, 2021

We checked your server. Seems like WCS is successfully updated to 5.2.983 and running:

Unfortunately we cannot test streaming because web interface ports 8444 and 8888 seems to be closed by security group, but server listens them

manikandan · Jul 15, 2021

Okay. I have a another server with a user data script which modifies the flashphoner.properties file, database.yml by downloading from S3 and replacing it in latest package conf folder.
On doing that, I am seeing unhealthy error in the service logs. Please check the last three lines

WCS server is actually running but I am concerned about the error (
FlashphonerWebCallServer: startingFlashphonerWebCallServer started, but is not healthy, please try to restart)

Max · Jul 16, 2021

manikandan said:
WCS server is actually running but I am concerned about the error (
FlashphonerWebCallServer: startingFlashphonerWebCallServer started, but is not healthy, please try to restart)

This message is displayed if WCS did not respond to health check query http://wcs:8081/health-check. Please check if WCS listens to all the ports necessary and web admin dashboard is accessible.
Ports can be checked by the following command:

Code:

sudo netstat -nlp | grep java

Also you can check if statistic page is displayed:

Code:

curl -s http://localhost:8081/?action=stats

If not, this means some misconfiguration.
You can also provide SSH access to this server, we will check.

manikandan · Jul 16, 2021

Max said:
http://wcs:8081/health-check

Returns 404

> sudo netstat -nlp | grep java
yes, all ports are open

>
curl -s http://localhost:8081/?action=stat
Yes. Its returning response

Shared the SSH access. Please check.

Max · Jul 16, 2021

The server is working, but you occasionally disabled healt checking endpoint, so it returns 404. Please replace the parameter in config

Code:

http_enable_paths=rest,action,shared,embed_player,empty

to

Code:

http_enable_paths=rest,action,shared,embed_player,empty,health-check

then restart WCS. In this case, health checking endpoint will be enabled again (and will return 200 OK if server is running successfully).

manikandan · Jul 16, 2021

Great. Its working now. Thank you Max.

ett · Jul 18, 2021

Hi Max.
I had a similar problem right after the update.
I think there is a problem with the update process.
I lost the log, but there was a syntax error such a ings: command not found.
The update process is rewriting bin/webcallserver.
At least in bash, you should not rewrite it while the shell script is running.

Max · Jul 19, 2021

ett said:
I had a similar problem right after the update.
I think there is a problem with the update process.

Please clarify from what version you've updated, and to what one (for example, from 5.2.780 to 5.2.993). We well check.

ett said:
The update process is rewriting bin/webcallserver.
At least in bash, you should not rewrite it while the shell script is running.

Please note that current webcallserver script instance exits right after starts a new one. And, if you're running it as a service (normally), a new instance will start as service systemctl start webcallserver too.
You can always update as described here stopping WCS manually before update.

ett · Jul 19, 2021

I tried it just now.
It is from 5.2.944 to 5.2.992 on AWS EC2 instance using ami-0e0f48424cc7236cf (5.2.944-systemd246)
"webcallserver update" is called from a shell script, not directly from the console.

Code:

+ systemctl stop webcallserver
+ /usr/local/FlashphonerWebCallServer/bin/webcallserver update 992 yes
>>> Version 5.2.992 is available, try to update
>>> Updating FlashphonerWebCallServer
>>> Downloading 5.2.992 build
yes: standard output: Broken pipe
yes: write error
>>> FlashphonerWebCallServer updated to 5.2.992

/usr/local/FlashphonerWebCallServer/bin/webcallserver: line 1757: Ip: command not found
/usr/local/FlashphonerWebCallServer/bin/webcallserver: line 1758: syntax error near unexpected token `fi'
/usr/local/FlashphonerWebCallServer/bin/webcallserver: line 1758: `  fi'

Max · Jul 20, 2021

ett said:
+ /usr/local/FlashphonerWebCallServer/bin/webcallserver update 992 yes

If you've stopped WCS before, yes is not needed.
And if yiou've stopped WCS before updating, it should not restart automatically. We raised the ticket WCS-3256 to check this.
Please install update as described here:

Code:

wget https://flashphoner.com/downloads/builds/WCS/5.2/FlashphonerWebCallServer-5.2.992.tar.gz
tar -xzf FlashphonerWebCallServer-5.2.992.tar.gz
sudo systemctl stop webcallserver 
cd FlashphonerWebCallServer-5.2.992
sudo ./install.sh
sudo systemctl start webcallserver

Max · Aug 24, 2021

Good day.
We fixed update issue since build 5.2.1015.

WCS service issue with AWS Markekplace AMI

manikandan

Member

Max

Administrator

manikandan

Member

Attachments

Max

Administrator

manikandan

Member

Max

Administrator

manikandan

Member

Max

Administrator

manikandan

Member

Max

Administrator

manikandan

Member

Max

Administrator

manikandan

Member

ett

Member

Max

Administrator

ett

Member

Max

Administrator

Max

Administrator