WebCallServer start not working with EFS mount

[Taylor]

Hello

We're having trouble deploying the latest version of the WebCallServer (5.2.944) on AWS.

When deploying a WebCallServer, we also mount Amazon's EFS to the records directory so that we can have multiple WebCallServers sharing the same network file system.

However when deploying the latest version it fails to restart the WCS successfully since after waiting a minute for the command to execute and continue, we cannot access the demo page. We can still ssh into the server but running top in the terminal window doesn't show 'java' as one of the active processes running.

This only seems to happen when a particular EFS is mounted:
Works fine when deploying without EFS.
Works fine when deploying with EFS (Total Size: ~50GiB).
Fails when deploying with EFS (Total Size: > 300GiB).

Regarding the deployment with EFS of 300GiB, it failed to finish restarting the WCS successfully, but then after about 15 minutes from executing the command we could then access the demo page and do stream recording fine.

We have also tested changing the record_dir property to "/usr/local/FlashphonerWebCallServer/records_efs" and creating a directory there and mounting it. This seems to work fine too (restarting WCS responds much faster).

---

We would greatly appreciate help in figuring out why restarting the WCS takes so long with EFS and if there's a solution that doesn't drastically change our architecture.

The record_dir is the best solution we currently have but since we need to access files in that mounted directory, that'll require deploying something new to access the recorded files from the EFS (and having someway to determine which EFS to fetch from) and thus not an ideal solution for us.

Configurations:
WCS Version: 5.2.944
AMI: Amazon Linux 2 AMI (SystemD246)
Instance Type: t3a.medium
EFS: Enabled
Flashphoner Properties:

stream_record_policy_template={streamName}
wss.port=443
ws.port=80
periodic_fir_request=true
record_audio_codec_channels=1
file_recorder_thread_pool_max_size=8

Java Memory Settings:

-Xms2g -Xmx2g
-XX:+UseConcMarkSweepGC
-XX:NewSize=256m

 

[Max]

Good day.
Please update WCS to the latest build 5.2.1040. Then, change the following parameter in /usr/local/FlashphonerWebCallServer/bin/setenv.sh from

WCS_NON_ROOT=true

to

WCS_NON_ROOT=false

Then test. If this does not help, please provide SSH access to the instance using this form.

 

[Taylor]

Hello Max

Yes the WCS_NON_ROOT=false fixed the issue, however there were odd issues when upgrading to 5.2.1040 before applying the fix, which I think are worth noting.

WCS 5.2.944 (EFS Setup) [WCS_NON_ROOT=true]:
Took ~3 minutes to access demo page after restarting.
Can record stream afterwards.
WCS 5.2.1040 (EFS Setup) [WCS_NON_ROOT=true]:
Couldn't access demo page after restarting, even after 30 minutes.
WCS 5.2.944 (No EFS) [WCS_NON_ROOT=true]:
Took ~7 seconds to access demo page after restarting.
Can record stream afterwards.
WCS 5.2.1040 (No EFS) [WCS_NON_ROOT=true]:
Took ~7 seconds to access demo page after restarting.
Unable to do stream recording afterwards.
WCS 5.2.944 (EFS Setup) [WCS_NON_ROOT=false]:
Took ~2 minutes to access demo page after restarting.
Can record stream afterwards.
WCS 5.2.1040 (EFS Setup) [WCS_NON_ROOT=false]:
Took ~7 seconds to access demo page after restarting.
Can record stream afterwards.
WCS 5.2.1040 (No EFS) [WCS_NON_ROOT=false]:
Took ~7 seconds to access demo page after restarting.
Can record stream afterwards.

WCS 5.2.1040 seems to work fine once the WCS_NON_ROOT is in place but with the 4th test above, when it wasn't used, I found it confusing how I could access the demo page but couldn't do a stream recording:

I tried testing on port 443 and 8443 just to check, but neither worked and I confirmed the flashphoner.properties were using 443 for the wss.port.

I'd also like to know how setting the WCS_NON_ROOT to false fixed the issue.

Thanks

 

[Max]

WCS 5.2.1040 seems to work fine once the WCS_NON_ROOT is in place but with the 4th test above, when it wasn't used, I found it confusing how I could access the demo page but couldn't do a stream recording:

Please check if WCS is listening port 443 in this case:

sudo netstat -nlp | grep java | grep 443

If not, collect the report as described here and send using this form or provide SSH access to the instance using this form.
The tweak with WCS_NON_ROOT=false makes WCS to start from root user and disables write privilegies checking on folders used by WCS. Seems like objects checking on EFS volume take a much time, so running as root is preferrable in this case.
To run 5.2.944 as root, you should additionally modify /etc/systemd/system/webcallserver.service file by replacing

User=flashphoner
Group=flashphoner

with

User=root
Group=root

 

[Taylor]

Hello Max

I ran the netstat command above and nothing was listening to port 443. I even tried 8443 but nothing was listed from that too.

I've send a report using the form you linked above.

Thanks

 

[Max]

I ran the netstat command above and nothing was listening to port 443. I even tried 8443 but nothing was listed from that too.

In startup.log file we see that WCS is starting in WCS_NON_ROOT=true mode:

[2021-09-27 01:58:21] INFO startWithSudo - Starting FlashphonerWebCallServer as user flashphoner

Please make sure that you've set WCS_NON_ROOT=false in /usr/local/FlashphonerWebCallServer/bin/setenv.sh file (this is not included in report), and user and group in /etc/systemd/system/webcallserver.service are set to root as recommended above. If nothing helps, provide SSH access to the server using this form.