Amazon AWS instance stopped working

[stather-it-consulting]

Hello

Suddenly, my Amazon AWS instance of Flashphoner stopped working, i cannot access https://<ip>:8888 anymore.

I recreated the instance and updated to the latest version. However, it is still not reachable, ssh to the server works however.
With netstat, i cannot see port 888 to be opened

---
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:sunrpc 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:ssh 0.0.0.0:* LISTEN
tcp 0 0 localhost:smtp 0.0.0.0:* LISTEN
tcp 0 64 ip-172-31-19-154.eu:ssh 185.69.236.65:52179 ESTABLISHED
tcp6 0 0 [::]:sunrpc [::]:* LISTEN
tcp6 0 0 [::]:ssh [::]:* LISTEN
tcp6 0 0 [::]:jetdirect [::]:* LISTEN
udp 0 0 0.0.0.0:bootpc 0.0.0.0:*
udp 0 0 0.0.0.0:sunrpc 0.0.0.0:*
udp 0 0 localhost:323 0.0.0.0:*
udp 0 0 0.0.0.0:netviewdm2 0.0.0.0:*
udp6 0 0 [::]:sunrpc [::]:*
udp6 0 0 localhost6:323 [::]:*
udp6 0 0 ip-172-31:dhcpv6-client [::]:*
udp6 0 0 [::]:netviewdm2 [::]:*
---

What can i do?

Best,
Thomas

 

[stather-it-consulting]

[ec2-user@ip-172-31-19-154 ~]$ systemctl status webcallserver
● webcallserver.service - Flashphoner WebCallServer
Loaded: loaded (/etc/systemd/system/webcallserver.service; enabled; vendor preset: disabled)
Active: active (exited) (Result: exit-code) since Wed 2021-06-02 09:04:02 UTC; 8min ago
Process: 3589 ExecStart=/bin/bash webcallserver start (code=exited, status=1/FAILURE)
Main PID: 3589 (code=exited, status=1/FAILURE)
CGroup: /system.slice/webcallserver.service

Jun 02 09:04:02 ip-172-31-19-154.eu-west-1.compute.internal systemd[1]: Started Flashphoner WebCallServer.
Jun 02 09:04:02 ip-172-31-19-154.eu-west-1.compute.internal systemd[1]: Starting Flashphoner WebCallServer...
Jun 02 09:04:04 ip-172-31-19-154.eu-west-1.compute.internal bash[3076]: FlashphonerWebCallServer: starting[ OK ]
Jun 02 09:04:08 ip-172-31-19-154.eu-west-1.compute.internal systemd[1]: webcallserver.service: main process exited, code=exited, status=1/FAILURE

I cannot even start it anymore

[ec2-user@ip-172-31-19-154 ~]$ systemctl start webcallserver
Failed to start webcallserver.service: The name org.freedesktop.PolicyKit1 was not provided by any .service files
See system logs and 'systemctl status webcallserver.service' for details.

 

[stather-it-consulting]

gc-core-2021-06-02_09-04.log holds
---
[2021-06-02T09:04:03.775+0000] Initializing The Z Garbage Collector
[2021-06-02T09:04:03.775+0000] Version: 14.0.1+7 (release)
[2021-06-02T09:04:03.775+0000] NUMA Support: Disabled
[2021-06-02T09:04:03.775+0000] CPUs: 4 total, 4 available
[2021-06-02T09:04:03.775+0000] Memory: 7470M
[2021-06-02T09:04:03.775+0000] Large Page Support: Disabled
[2021-06-02T09:04:03.775+0000] Medium Page Size: 32M
[2021-06-02T09:04:03.775+0000] Workers: 3 parallel, 1 concurrent
[2021-06-02T09:04:03.775+0000] Address Space Type: Contiguous/Unrestricted/Complete
[2021-06-02T09:04:03.775+0000] Address Space Size: 262144M x 3 = 786432M
[2021-06-02T09:04:03.775+0000] Heap backed by file: /memfd:java_heap
[2021-06-02T09:04:03.775+0000] Min Capacity: 16384M
[2021-06-02T09:04:03.775+0000] Initial Capacity: 16384M
[2021-06-02T09:04:03.775+0000] Max Capacity: 16384M
[2021-06-02T09:04:03.775+0000] Max Reserve: 38M
[2021-06-02T09:04:03.775+0000] Pre-touch: Disabled
[2021-06-02T09:04:03.775+0000] Available space on backing filesystem: N/A
[2021-06-02T09:04:05.387+0000] Failed to commit memory (No space left on device)
[2021-06-02T09:04:06.743+0000] Failed to commit memory (No space left on device)
[2021-06-02T09:04:07.617+0000] Failed to commit memory (No space left on device)
[2021-06-02T09:04:07.724+0000] Failed to commit memory (No space left on device)
[2021-06-02T09:04:07.725+0000] Failed to commit memory (No space left on device)
[2021-06-02T09:04:07.727+0000] Failed to commit memory (No space left on device)
[2021-06-02T09:04:07.727+0000] Failed to commit memory (No space left on device)
[2021-06-02T09:04:07.728+0000] Failed to allocate initial Java heap (16384M)
---

This is a c4.large instance (it worked before)

 

[Max]

Good day.
Please re-create instance from scratch using latest AWS Marketplace image (build 5.2.944)

The problem with startup on weak server configuration is fixed in this image.

 

[stather-it-consulting]

I only have 5.2.940 available in the dropdown

 

[stather-it-consulting]

I had to select "view all version on AWS marketplace" to be able to launch 5.2.944

 

[stather-it-consulting]

It is still not working.

-I recreated the instance from scratch
-I am able to ssh to my elastic ip
-I am not able to open https://<ip>:8888

 

[stather-it-consulting]

Like i said it worked before (in 5-5-2-780) but then stopped working this morning. So i updated the instance and now

The flashphoner log says
---
11:51:41,453 INFO WSServerHandler - WSS-pool-23-thread-1 Orgign: null
11:51:41,458 WARN WSServerHandler - WSS-pool-23-thread-1 Close channel [id: 0x7e3b4ec6, /74.82.47.3:55334 => /172.31.16.47:8443] because: org.jboss.netty.handler.codec.http.websocketx.WebSocketHandshakeException 'not a WebSocket handshake request: missing upgrade'
11:51:41,459 INFO WSClients - WSClientsKeepaliveThread-79 isKeepAliveProtoEnabled websocket true
11:51:41,459 INFO WSClients - WSClientsKeepaliveThread-79 Start WebSocket ping thread: WSClientsKeepaliveThread-79 interval: 5000 algorithm: HIGH_LEVEL
11:51:56,656 WARN WSServerHandler - WSS-pool-23-thread-1 Close channel [id: 0x207a98ab, /74.82.47.3:49204 => /172.31.16.47:8443] because: javax.net.ssl.SSLHandshakeException 'Client requested protocol SSLv3 is not enabled or supported in server context'
11:52:02,735 WARN WSServerHandler - WSS-pool-23-thread-1 Close channel [id: 0x10e190f5, /74.82.47.3:49130 => /172.31.16.47:8443] because: javax.net.ssl.SSLHandshakeException 'no cipher suites in common'
11:52:13,626 WARN WSServerHandler - WSS-pool-23-thread-1 Close channel [id: 0x0240adaf, /74.82.47.3:53312 => /172.31.16.47:8443] because: java.net.SocketException 'Connection reset'
11:52:13,627 WARN WSServerHandler - WSS-pool-23-thread-1 Close channel [id: 0x0240adaf, /74.82.47.3:53312 => /172.31.16.47:8443] because: java.io.IOException 'Broken pipe'
11:52:16,028 WARN WSServerHandler - WSS-pool-23-thread-1 Close channel [id: 0x4ccb6340, /74.82.47.3:34254 => /172.31.16.47:8443] because: javax.net.ssl.SSLHandshakeException 'no cipher suites in common'
11:53:18,599 WARN WSServerHandler - WSS-pool-23-thread-1 Close channel [id: 0x4528f669, /74.82.47.3:47286 => /172.31.16.47:8443] because: java.net.SocketException 'Connection reset'
---

Please help me

 

[Max]

Did you do a backup?

To get this working you need a valid domain name FQDN and imported SSL certificates for this domain.
If you have a backup you can just copy WCS_HOME/conf/wss.jks from your previous installation and restart server.
This file wss.jks keeps imported SSL certificates. Anyway you have to connect to the server using FQDN, i.e. wss://test.mycompany.com:8443 and connection must be secure (with SSL certs imported to the server).

Another option for Amazon is load balancer. I.e. if AWS Load balancer has FQDN URL test.mycompany.com:443 > instance:8443 (having WCS instance 8443 in the background. In such a case WCS instance can be insecure).

Regarding your initial issue (No space left on device)
If you have a backup, you can try your old instance again and check if disk is full.

 

[stather-it-consulting]

I have no backup, the old instance stopped working so i recreated it and nowthe new instance is not working after fresh creation

I have the cert and the private key, please tell me how to import them on the command line.
Another question is why isn't there a default certificate installed like in the old instance?
Because in the old instance i upladed my cert and private key over the webui (after initial install i could access the webui).

 

[Max]

You can import cert the same way via Web UI.
Sign in to https://instance-ip:8444 as admin with password: 12345
Here 12345 is your Amazon AWS instance ID.
Once you logged, click Security > Certificates and upload certificate and private key.
Docs: https://docs.flashphoner.com/displa...ketSSL-ImportSSLcertificatesusingwebinterface
If you have any troubles with importing certs, please send cert files, private key, and SSH access to the server using this private form.
We'll import certs for you.

Each pair of certs (cert + private key) is unique and issued for particular domain, i.e. stream.mycompany.com.
We are not a certification center or SSL provider (CA). That's why we can't just put valid FQDN certs to installation package.

 

[stather-it-consulting]

Hi.

Ok this worked for me and i could install my custom certificates

 

[stather-it-consulting]

Did the port for the webui and when used for embedding change from 8888 to 8444? Because in the past i only used 8888
The websocket port is still 8443, right?

 

[stather-it-consulting]

Ok another problem, when i try to stream media using "Media Devices" i can connect to the websocket but every time i try to start the stream it says "FAILED"

 

[stather-it-consulting]

I did the following to set the ip address
---
ssh -p 2001 admin@localhost
<provide password>

show node-settings
show node-setting ip
update node-setting --value <amazon elastic ip>
save node-settings

However, i cannot stop and start the instance from within ssh

[ec2-user@ip-172-31-16-47 ~]$ systemctl stop webcallserver
Failed to stop webcallserver.service: The name org.freedesktop.PolicyKit1 was not provided by any .service files
See system logs and 'systemctl status webcallserver.service' for details.

[ec2-user@ip-172-31-16-47 ~]$ systemctl restart webcallserver
Failed to restart webcallserver.service: The name org.freedesktop.PolicyKit1 was not provided by any .service files
See system logs and 'systemctl status webcallserver.service' for details.

 

[Max]

Did the port for the webui and when used for embedding change from 8888 to 8444? Because in the past i only used 8888

Yes, the default HTTPS port is 8444/tcp. But 8888/tcp is also supported for backward compatibility.

The websocket port is still 8443, right?

Yes, 8443/tcp is still a default secure websocket port (and 8080/tcp for unsecure one)

Ok another problem, when i try to stream media using "Media Devices" i can connect to the websocket but every time i try to start the stream it says "FAILED"

Please check if you're opened media ports (30000-33000/udp) in instance security group.

However, i cannot stop and start the instance from within ssh

Please use sudo to perform all the actions requiring root privilegies

sudo systemctl restart webcallserver

 

[stather-it-consulting]

Thanks a lot for all your help. Now i got it working in the end (after restarting the Flashphoner service it took the new ip address set previously, i think).