Issues Updating WCS on AWS

[Taylor]

Hello

We've been trying to install the latest WCS build 5.2.2013 onto our AWS platform but have been experiencing a few issues.

Our process is to launch a new EC2 instance using the latest WCS image on AWS marketplace, currently at 5.2.1482, download and install the latest build onto the server and then make some updates to the flashphoner.properties, wcs-core.properties and bin/setenv.sh before calling sudo service webcallserver restart.

But when we try calling restart, it fails to restart the server and returns with this:

[ec2-user@ip-10-0-7-173 ~]$ sudo service webcallserver restart
Redirecting to /bin/systemctl restart webcallserver.service
Job for webcallserver.service failed because a timeout was exceeded. See "systemctl status webcallserver.service" and "journalctl -xe" for details.

And running journalctl reveals it fails to start the WebCallServer but doesn't explain why as far as we can tell:

...
Jul 09 03:57:30 ip-10-0-1-95.ap-southeast-2.compute.internal systemd[1]: Can't open PID file /usr/local/FlashphonerWebCallServer/bin/FlashphonerMainWebCallServer.pid (yet?) after start: No such file or director
Jul 09 03:57:30 ip-10-0-1-95.ap-southeast-2.compute.internal systemd[1]: Failed to start Flashphoner WebCallServer.
Jul 09 03:57:30 ip-10-0-1-95.ap-southeast-2.compute.internal systemd[1]: Unit webcallserver.service entered failed state.
Jul 09 03:57:30 ip-10-0-1-95.ap-southeast-2.compute.internal systemd[1]: webcallserver.service failed.
Jul 09 03:57:30 ip-10-0-1-95.ap-southeast-2.compute.internal systemd[1]: Removed slice User Slice of flashphoner.
Jul 09 03:57:35 ip-10-0-1-95.ap-southeast-2.compute.internal systemd[1]: webcallserver.service holdoff time over, scheduling restart.
Jul 09 03:57:35 ip-10-0-1-95.ap-southeast-2.compute.internal systemd[1]: Cannot add dependency job for unit systemd-sysusers.service, ignoring: Unit is not loaded properly: Invalid argument.
Jul 09 03:57:35 ip-10-0-1-95.ap-southeast-2.compute.internal systemd[1]: Cannot add dependency job for unit systemd-boot-system-token.service, ignoring: Unit is not loaded properly: Invalid argument.
Jul 09 03:57:35 ip-10-0-1-95.ap-southeast-2.compute.internal systemd[1]: Stopped Flashphoner WebCallServer.
Jul 09 03:57:35 ip-10-0-1-95.ap-southeast-2.compute.internal systemd[1]: Starting Flashphoner WebCallServer...
Jul 09 03:57:35 ip-10-0-1-95.ap-southeast-2.compute.internal sudo[36607]:     root : TTY=unknown ; PWD=/usr/local/FlashphonerWebCallServer-5.2.2013/bin ; USER=flashphoner ; COMMAND=/bin/echo#040-e#040[2024-07-0
Jul 09 03:57:35 ip-10-0-1-95.ap-southeast-2.compute.internal systemd[1]: Created slice User Slice of flashphoner.
Jul 09 03:57:35 ip-10-0-1-95.ap-southeast-2.compute.internal systemd[1]: Started Session c34 of user flashphoner.
Jul 09 03:57:35 ip-10-0-1-95.ap-southeast-2.compute.internal sudo[36607]: pam_unix(sudo:session): session opened for user flashphoner by (uid=0)
Jul 09 03:57:35 ip-10-0-1-95.ap-southeast-2.compute.internal sudo[36607]: pam_unix(sudo:session): session closed for user flashphoner
Jul 09 03:57:35 ip-10-0-1-95.ap-southeast-2.compute.internal bash[36544]: FlashphonerWebCallServer is already running
Jul 09 03:57:35 ip-10-0-1-95.ap-southeast-2.compute.internal systemd[1]: Removed slice User Slice of flashphoner.
Jul 09 03:57:35 ip-10-0-1-95.ap-southeast-2.compute.internal sudo[36624]:     root : TTY=unknown ; PWD=/usr/local/FlashphonerWebCallServer-5.2.2013/bin ; USER=flashphoner ; COMMAND=/bin/echo#040-e#040[2024-07-0
Jul 09 03:57:35 ip-10-0-1-95.ap-southeast-2.compute.internal systemd[1]: Created slice User Slice of flashphoner.
Jul 09 03:57:35 ip-10-0-1-95.ap-southeast-2.compute.internal systemd[1]: Started Session c35 of user flashphoner.
Jul 09 03:57:35 ip-10-0-1-95.ap-southeast-2.compute.internal sudo[36624]: pam_unix(sudo:session): session opened for user flashphoner by (uid=0)
Jul 09 03:57:35 ip-10-0-1-95.ap-southeast-2.compute.internal sudo[36624]: pam_unix(sudo:session): session closed for user flashphoner
Jul 09 03:57:35 ip-10-0-1-95.ap-southeast-2.compute.internal systemd[1]: Can't open PID file /usr/local/FlashphonerWebCallServer/bin/FlashphonerMainWebCallServer.pid (yet?) after start: No such file or director
Jul 09 03:57:35 ip-10-0-1-95.ap-southeast-2.compute.internal systemd[1]: Failed to start Flashphoner WebCallServer.
Jul 09 03:57:35 ip-10-0-1-95.ap-southeast-2.compute.internal systemd[1]: Unit webcallserver.service entered failed state.
...

In addition, any attempt to do video streaming gets disconnected in the middle of the stream (usually within 30 seconds).

The same error also happens with the command sudo systemctl restart webcallserver, but when we run sudo /usr/local/FlashphonerWebCallServer/bin/webcallserver restart it seems to correctly restart the WebCallServer and we can do video streaming without getting disconnected.

This unfortunately isn't an effective solution because anytime we need to stop and start the EC2 instance (say bringing it up only when demand calls for it) it would have the same issue with being disconnected in the middle of streaming, and it doesn't explain why the restart commands are broken in the first place.



What we do know is that this issue is due to upgrading to a newer version AND due to in part of our changes to the properties/environment.

Factor 1: The WCS version this started happening in occurs between versions 5.2.1504 and 5.2.1603, as in when we upgrade to 5.2.1504 and then make changes to the properties/environment, the restart command works and we can successfully stream videos. However this is many versions behind the latest and is not ideal to just settle at version 5.2.1504.

Factor 2: It's due to one or more of our changes to the properties/environment. We've tried removing most of the changes where we can (but kept our changes on flashphoner.properties due to the WebSocket ports) and found the media server can correctly restart, but we are unable to do video streaming as it returns with a "Failed to Connect" error upon starting on the 'Stream Recording' page on the Media Server.

conf/flashphoner.properties

stream_record_policy_template={streamName}
wss.port                =443
ws.port                 =80
periodic_fir_request=true
record_audio_codec_channels=1
file_recorder_thread_pool_max_size=8

/usr/local/FlashphonerWebCallServer/bin/setenv.sh
WCS_NON_ROOT=false
(Note: We believe this is one of the changes that causes the issue, but not entirely sure. This was originally added in to handle EFS setup back in: https://forum.flashphoner.com/threads/webcallserver-start-not-working-with-efs-mount.13926/#post-29709)

conf/wcs-core.properties

-Xms2g -Xmx2g
-XX:+UseConcMarkSweepGC
-XX:NewSize=256m

Other updates include:

• Updating /etc/hostname and /etc/hosts with our hostname and loopback address.
• Mounting EFS to where the videos are recorded.
• Updating the demo password for the account.

Since we normally upgrade the WCS version manually (using wget, unzipping and running install.sh), we tried running the bin/webcallserver update command to update instead.

• If we call it before we make changes to the properties/environment, it runs fine but has same issue once we make those property/environment changes.
• If we call it after we make changes to the properties/environment, it fails to run and seems to stop just after downloading the zip to the /tmp folder.

We're not sure if this is related to but one other thing we've noticed is that when we log on to the media server after upgrading, this error appears in the console logs:

Any help with understanding and fixing this issue would be greatly appreciated.

Cheers

 

[Max]

Good day.

/usr/local/FlashphonerWebCallServer/bin/setenv.sh
WCS_NON_ROOT=false

That's the cause of the issue. You must change also /etc/systemd/system/webcallserver.service file when switching to root mode. So we recommend the following:
1. Download and install latest build
2. Change flashphoner.properties and wcs-core.properties
3. Use the following command to switch to root mode (see Switching launch mode)

sudo /usr/local/FlashphonerWebCallServer/bin/webcallserver set-root-mode enable

The command will apply a necessary changes
Then, WCS should start normally.

We're not sure if this is related to but one other thing we've noticed is that when we log on to the media server after upgrading, this error appears in the console logs:

This is not the error. In the latest WebSDK builds, the methods were added to configure server ports to connect using custom SERVER_PORTS object, for example:

/**
 * Server ports configuration for load testing
 *
 */
const SERVER_PORTS = {
    http: 8081,
    https: 8444,
    legacy: {
        http: 9091,
        https: 8888
    },
    ws: 8080,
    wss: 8443,
    hls: {
        http: 8082,
        https: 8445
    }
}

If the SERVER_PORTS object is not defined, the default server ports are used and the message is displayed to browser console.
See console.html and ports.js on GitHub for reference.

 

[Taylor]

Hi Max

Thank you for the quick reply. I've applied the change and can confirm our media server now works.