websocket connection increase after few days

[Stanley]

Hi Max,
I notice our Flashphoner server v5.1.3367 stream_websocket_out and connections_websocket will increase to 1000 after few days running. We doesn't have that many concurrent users running WSPlayer. Is there any way to kill the idle connection timeout from server side?

 

[Max]

Hello

You can use REST

/rest-api/connection/find_all
/rest-api/connection/find
/rest-api/terminate

methods to manage websocket connections
See also https://docs.flashphoner.com/display/WCS5EN/Examples

From your report it looks like a leak.
Please zip and share logs for last few days. We will check if it indicates any issues.
WCS_HOME/logs/server_logs
support@flashphoner.com

Please also provide your configuration WCS_HOME/conf folder.

 

[Stanley]

Hi Max, I've submitted the log yesterday. Do you have any findings ?

 

[Max]

Hello

We do not see any errors in logs.
This is what you can check

1. Check the real number of websocket connections.

lsof -nPi TCP:8443

2. Check number of websocket connections via REST API
http://host:9091/rest-api/connection/find_all

If you found a zombie connection you can see its IP address and port.
Try to find this connection in logs.

WCS_HOME/logs/cdr/conndr.log
WCS_HOME/logs/server_logs

If it is found, please share logs again with list of leaking connections (IP and port).

You can also add debug log for more information about Websocket connections.
log4j.properties

log4j.logger.WSClients=DEBUG

 

[Max]

Hello
Could you please
1. Re-send client logs
2. Send flashphoner.properties
3. Check keep_alive property in conf file, it should not to be disabled (https://docs.flashphoner.com/display/WCS5EN/Settings+file+flashphoner.properties)

 

[Stanley]

Hi Max,
keep_alive.algorithm =INTERNAL
keep_alive.peer_interval =2000
keep_alive.server_interval =2000
keep_alive.probes =2

I've tried INTERNAL and HIGH_LEVEL problem persists..

 

[Max]

What about connections? We didn't receive lsof output:

lsof -nPi TCP:8443

and /connection/find_all
Please check out previous message: https://forum.flashphoner.com/threads/websocket-connection-increase-after-few-days.11594/#post-17199

 

[Stanley]

Hi Max,
I've sent you the server logs as per your request on last Saturday.

 

[Max]

Hello
We have checked logs. Thanks for the report.
Form the logs, find_all, lsof we don't see a lot of real connections.
So it looks like issue with counter of Websocket streams.
Try
http://host:9091/rest-api/stream/find_all
And count PLAYING streams.
If number of playback streams equals stream_websocket_out, it can point to a leak.
For now it looks like minor issue with connection counter.

Anyway, we will prepare a new build with debug information to localize the issue.
We will inform you once it is ready.

 

[Max]

Hello
We have released build 3493, which contains additional logs. Please, update to latest build. For next investigations we need client logs and heap dump from your server.
To enable logs add this to flashphoner.properties:

enable_extended_logs=true
client_log_level=debug

To make heap dump follow this:
1. Exec jps and find out PID of Server process:

# jps
5523 Jps
5038 wcs_manager-1.0.jar
5121 Server

2. Exec jmap to get dump:

jmap -dump:file=/tmp/server.hprof 5121

3. Arch hprof file, it may have over 3-4 Gb size. Share dump with Google Drive or similar.