WCS service issue with AWS Markekplace AMI

Hello,

I launched WCS server using AWS marketplace AMI https://aws.amazon.com/marketplace/reviews/reviews-list/B07ZVRPH5G and done similar to this setup.

Also updated the WCS package to latest version using the bash script in user data.

Upon launching the instance, when I checked with the webcallserver status I noticed some errors in the service


Code:
[ec2-user@ip-172-31-70-30 ~]$ sudo service webcallserver status -l
Redirecting to /bin/systemctl status  -l webcallserver.service
● webcallserver.service - Flashphoner WebCallServer
   Loaded: loaded (/etc/systemd/system/webcallserver.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2021-07-14 16:27:59 UTC; 14s ago
  Process: 4306 ExecStop=/bin/bash webcallserver stop (code=exited, status=0/SUCCESS)
 Main PID: 4995 (java)
   CGroup: /system.slice/webcallserver.service
           ‣ 4995 java -Xmx512M -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=50999 -Dcom.sun.management.jmxremote.host=localhost -Djava.rmi.server.hostname=localhost -XX:ErrorFile=/usr/local/FlashphonerWebCallServer/logs/error%p.log -Xlog:gc*:/usr/local/FlashphonerWebCallServer/logs/gc-core-2021-07-14_16-27.log:time -XX:+ExplicitGCInvokesConcurrent -Dsun.rmi.dgc.client.gcInterval=36000000000 -Dsun.rmi.dgc.server.gcInterval=36000000000 -Dcom.flashphoner.fms.AppHome=/usr/local/FlashphonerWebCallServer -Djava.library.path=/usr/local/FlashphonerWebCallServer/lib/so:/usr/local/FlashphonerWebCallServer/lib -DWCS_NON_ROOT=true -DsessionDebugEnabled=false -Djdk.tls.client.protocols="TLSv1,TLSv1.1,TLSv1.2" -cp /usr/local/FlashphonerWebCallServer/lib/* com.flashphoner.server.Server

Jul 14 16:28:00 ip-172-31-70-30.ec2.internal sudo[4927]:     root : TTY=unknown ; PWD=/usr/local/FlashphonerWebCallServer-5.2.983-39b59ee9dba7df2a6163fdc2eb92aeb72bb90ebb/bin ; USER=flashphoner ; COMMAND=/bin/env LD_LIBRARY_PATH=/usr/local/FlashphonerWebCallServer/lib/so java -Xmx512M -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=50999 -Dcom.sun.management.jmxremote.host=localhost -Djava.rmi.server.hostname=localhost -XX:ErrorFile=/usr/local/FlashphonerWebCallServer/logs/error%p.log -Xlog:gc*:/usr/local/FlashphonerWebCallServer/logs/gc-core-2021-07-14_16-27.log:time -XX:+ExplicitGCInvokesConcurrent -Dsun.rmi.dgc.client.gcInterval=36000000000 -Dsun.rmi.dgc.server.gcInterval=36000000000 -Dcom.flashphoner.fms.AppHome=/usr/local/FlashphonerWebCallServer -cp /usr/local/FlashphonerWebCallServer/lib/*
Jul 14 16:28:00 ip-172-31-70-30.ec2.internal sudo[4927]:     root : (command continued) com.flashphoner.server.license.ExpiredUpdates
Jul 14 16:28:02 ip-172-31-70-30.ec2.internal sudo[4979]:     root : TTY=unknown ; PWD=/usr/local/FlashphonerWebCallServer-5.2.983-39b59ee9dba7df2a6163fdc2eb92aeb72bb90ebb/bin ; USER=flashphoner ; COMMAND=/bin/echo -e [2021-07-14 16:28:02] INFO startWithSudo - Starting FlashphonerWebCallServer as user flashphoner
Jul 14 16:28:02 ip-172-31-70-30.ec2.internal sudo[4981]:     root : TTY=unknown ; PWD=/usr/local/FlashphonerWebCallServer-5.2.983-39b59ee9dba7df2a6163fdc2eb92aeb72bb90ebb/bin ; USER=flashphoner ; COMMAND=/bin/env LD_LIBRARY_PATH=/usr/local/FlashphonerWebCallServer/lib/so java -Xmx512M -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=50999 -Dcom.sun.management.jmxremote.host=localhost -Djava.rmi.server.hostname=localhost -XX:ErrorFile=/usr/local/FlashphonerWebCallServer/logs/error%p.log -Xlog:gc*:/usr/local/FlashphonerWebCallServer/logs/gc-core-2021-07-14_16-27.log:time -XX:+ExplicitGCInvokesConcurrent -Dsun.rmi.dgc.client.gcInterval=36000000000 -Dsun.rmi.dgc.server.gcInterval=36000000000 -Dcom.flashphoner.fms.AppHome=/usr/local/FlashphonerWebCallServer
Jul 14 16:28:02 ip-172-31-70-30.ec2.internal sudo[4981]:     root : (command continued) -Djava.library.path=/usr/local/FlashphonerWebCallServer/lib/so:/usr/local/FlashphonerWebCallServer/lib -DWCS_NON_ROOT=true -DsessionDebugEnabled=false -Djdk.tls.client.protocols="TLSv1,TLSv1.1,TLSv1.2" -cp /usr/local/FlashphonerWebCallServer/lib/* com.flashphoner.server.Server
Jul 14 16:28:04 ip-172-31-70-30.ec2.internal sudo[5057]:     root : TTY=unknown ; PWD=/usr/local/FlashphonerWebCallServer-5.2.983-39b59ee9dba7df2a6163fdc2eb92aeb72bb90ebb/bin ; USER=flashphoner ; COMMAND=/bin/touch /usr/local/FlashphonerWebCallServer/bin/FlashphonerWebCallServer
Jul 14 16:28:10 ip-172-31-70-30.ec2.internal sudo[5143]:     root : TTY=unknown ; PWD=/usr/local/FlashphonerWebCallServer-5.2.983-39b59ee9dba7df2a6163fdc2eb92aeb72bb90ebb/bin ; USER=flashphoner ; COMMAND=/bin/echo -e [2021-07-14 16:28:10] ERROR start - FlashphonerWebCallServer started, but is not healthy, please try to restart
Jul 14 16:28:10 ip-172-31-70-30.ec2.internal bash[4401]: FlashphonerWebCallServer: startingFlashphonerWebCallServer started, but is not healthy, please try to restart
Jul 14 16:28:10 ip-172-31-70-30.ec2.internal bash[4401]: [FAILED]
Jul 14 16:28:10 ip-172-31-70-30.ec2.internal systemd[1]: webcallserver.service: Supervising process 4995 which is not our child. We'll most likely not notice when it exits.
it seems some kind of permission issue when running service start command as root.

Let me know how it can be fixed?
 

Max

Administrator
Staff member
Good day.
Please try the following
Code:
sudo /usr/local/FlashphonerWebCallServer/webcallserver set-permissions
sudo systemctl restart webcallserver
 
sudo /usr/local/FlashphonerWebCallServer/webcallserver set-permissions
I think its sudo /usr/local/FlashphonerWebCallServer/bin/webcallserver set-permissions


Even after setting the permissions, still it has same error as above mentioned in the ticket. I have attached the script file

I am planning to use Marketplace AMI setup. Hoping to get this fixed.
 

Attachments

Max

Administrator
Staff member
We cannot reproduce the issue starting a new instance from latest Marketplace AMI (5.2.944-systemd246) with your userdata script: WCS is started successfully without any permission fixes.
Please provide SSH acceess to the problem instance using this form, we will check
 

Max

Administrator
Staff member
We checked your server. Seems like WCS is successfully updated to 5.2.983 and running:
1626343132901.png

1626343179300.png

Unfortunately we cannot test streaming because web interface ports 8444 and 8888 seems to be closed by security group, but server listens them
1626343370067.png
 
Okay. I have a another server with a user data script which modifies the flashphoner.properties file, database.yml by downloading from S3 and replacing it in latest package conf folder.
On doing that, I am seeing unhealthy error in the service logs. Please check the last three lines
1626346851357.png


WCS server is actually running but I am concerned about the error (
FlashphonerWebCallServer: startingFlashphonerWebCallServer started, but is not healthy, please try to restart)
 

Max

Administrator
Staff member
WCS server is actually running but I am concerned about the error (
FlashphonerWebCallServer: startingFlashphonerWebCallServer started, but is not healthy, please try to restart)
This message is displayed if WCS did not respond to health check query http://wcs:8081/health-check. Please check if WCS listens to all the ports necessary and web admin dashboard is accessible.
Ports can be checked by the following command:
Code:
sudo netstat -nlp | grep java
Also you can check if statistic page is displayed:
Code:
curl -s http://localhost:8081/?action=stats
If not, this means some misconfiguration.
You can also provide SSH access to this server, we will check.
 

Max

Administrator
Staff member
The server is working, but you occasionally disabled healt checking endpoint, so it returns 404. Please replace the parameter in config
Code:
http_enable_paths=rest,action,shared,embed_player,empty
to
Code:
http_enable_paths=rest,action,shared,embed_player,empty,health-check
then restart WCS. In this case, health checking endpoint will be enabled again (and will return 200 OK if server is running successfully).
 

ett

Member
Hi Max.
I had a similar problem right after the update.
I think there is a problem with the update process.
I lost the log, but there was a syntax error such a ings: command not found.
The update process is rewriting bin/webcallserver.
At least in bash, you should not rewrite it while the shell script is running.
 

Max

Administrator
Staff member
I had a similar problem right after the update.
I think there is a problem with the update process.
Please clarify from what version you've updated, and to what one (for example, from 5.2.780 to 5.2.993). We well check.
The update process is rewriting bin/webcallserver.
At least in bash, you should not rewrite it while the shell script is running.
Please note that current webcallserver script instance exits right after starts a new one. And, if you're running it as a service (normally), a new instance will start as service systemctl start webcallserver too.
You can always update as described here stopping WCS manually before update.
 

ett

Member
I tried it just now.
It is from 5.2.944 to 5.2.992 on AWS EC2 instance using ami-0e0f48424cc7236cf (5.2.944-systemd246)
"webcallserver update" is called from a shell script, not directly from the console.
Code:
+ systemctl stop webcallserver
+ /usr/local/FlashphonerWebCallServer/bin/webcallserver update 992 yes
>>> Version 5.2.992 is available, try to update
>>> Updating FlashphonerWebCallServer
>>> Downloading 5.2.992 build
yes: standard output: Broken pipe
yes: write error
>>> FlashphonerWebCallServer updated to 5.2.992

/usr/local/FlashphonerWebCallServer/bin/webcallserver: line 1757: Ip: command not found
/usr/local/FlashphonerWebCallServer/bin/webcallserver: line 1758: syntax error near unexpected token `fi'
/usr/local/FlashphonerWebCallServer/bin/webcallserver: line 1758: `  fi'
 

Max

Administrator
Staff member
+ /usr/local/FlashphonerWebCallServer/bin/webcallserver update 992 yes
If you've stopped WCS before, yes is not needed.
And if yiou've stopped WCS before updating, it should not restart automatically. We raised the ticket WCS-3256 to check this.
Please install update as described here:
Code:
wget https://flashphoner.com/downloads/builds/WCS/5.2/FlashphonerWebCallServer-5.2.992.tar.gz
tar -xzf FlashphonerWebCallServer-5.2.992.tar.gz
sudo systemctl stop webcallserver 
cd FlashphonerWebCallServer-5.2.992
sudo ./install.sh
sudo systemctl start webcallserver
 
Top