Connection status is showing many many connections

Max

Administrator
Staff member
Please share your number of build and server.properties and flashphoner.properties configs
We can't reproduce this with latest 2101 version.
If you see grow of these connections in real-time, please send your server_logs/flashphoner.log to logs@flashphoner.com. We will check.
 

phil

New Member
Please share your number of build and server.properties and flashphoner.properties configs
We can't reproduce this with latest 2101 version.
If you see grow of these connections in real-time, please send your server_logs/flashphoner.log to logs@flashphoner.com. We will check.
Thank you Max. I'm using build 2075. The configs are attached. Thank you.

Please note that "stream stats" on ":8081/?action=stat" page do release connections, but "Connection stats" do not.

Thank you
 

Attachments

Max

Administrator
Staff member
Your connections are never being released because you turned off the keep alive check for all connections:
Code:
keep_alive.algorithm=NONE
in server.properties
try to set this to
Code:
keep_alive.algorithm=INTERNAL
then restart wcs server
Code:
service webcallserver restart
 

phil

New Member
Your connections are never being released because you turned off the keep alive check for all connections:
Code:
keep_alive.algorithm=NONE
in server.properties
try to set this to
Code:
keep_alive.algorithm=INTERNAL
then restart wcs server
Code:
service webcallserver restart

Thanks Max. However, making it INTERNAL makes FFMPEG not able to send video streams to WCS.
What do the numbers mean ?
Will WCS shut down if the numbers are too high?
Is it total connections of all old connections?
 

Max

Administrator
Staff member
It is not released connections and such connections eat up your system memory.
WCS can't release this connection because it assume that connection is still alive if keep alives are disabled.
Thanks Max. However, making it INTERNAL makes ffmpeg not able to send video streams to WCS.
According our tests with latest ffmpeg it works well.
Could you please describe your ffmpeg configuration and what exactly does not work when keep alives are enabled?
We recently tested ffmpeg streaming with latest builds and it works properly using such command:
Code:
ffmpeg -re -i /tmp/VIDEO2findingnews.mp4 -acodec aac -vcodec libx264 -f flv -ar 44100 -rtmp_conn "O:1 NS:appKey:flashStreamingApp NS:name:12121212 NS:stream:12121212 NO:custom:O:1 NS:auth:22222222 NS:stream:3333333 O:0 O:0" rtmp://192.168.33.200:1935/12121212
or
Code:
ffmpeg -re -i /tmp/VIDEO2findingnews.mp4 -acodec aac -vcodec libx264 -f flv -ar 44100 rtmp://192.168.33.200:1935/12121212
So it should work.
If it does not work for you with latest build please describe your testing process in detail.
 

Max

Administrator
Staff member
You reported about memory leak here
It seems your leak was caused by this setting
Code:
keep_alive.algorithm=NONE
Try to set this to INTERNAL or HIGH_LEVEL.
It works in our tests with ffmpeg.
 

phil

New Member
You reported about memory leak here
It seems your leak was caused by this setting
Code:
keep_alive.algorithm=NONE
Try to set this to INTERNAL or HIGH_LEVEL.
It works in our tests with ffmpeg.
Thank you Max. The memory leak issue is fixed with build 2075. I was using 2036.
May I know what would happen if the number of connections goes too high?
 

Max

Administrator
Staff member
May I know what would happen if the number of connections goes too high?
Most likely it will cause Out Of Memory issue on a long-term period. It depends how many resources this connection holds.
 

phil

New Member
Most likely it will cause Out Of Memory issue on a long-term period. It depends how many resources this connection holds.
Thanks Max. If I don't set that to NONE, then FFMPEG has trouble sending streams to WCS because the previous connection session is kept if there's some lost connection.

The memory does grow a little bit, but it looks fine to me. Does the number of connections mean the total connections including all past connections ? If that is the case, then I don't think it is an issue.
 

Max

Administrator
Staff member
Does the number of connections mean the total connections including all past connections ?
No. It is not just a counter of past connections.
If connections=2561, this means WCS has 2561 objects, each of which represents the client connection.
These object may take a tiny of memory or may take a lot of memory. It depends, how many of internal references and threads it holds.
We will try to make the same test with keep alive NONE and check the heap utilization to answer your question.
 

Max

Administrator
Staff member
Hello
1. According our investigation,
Each of such connections takes around of 200 bytes, if it was not publish/play in this connection.
Each of such connections takes around of 30 k bytes, if it had publish/play in this connection.
2. We have found a bug with setting
Code:
keep_alive.algorithm=NONE
When keep alives are disabled, Websocket connection is not released after page reload (for example in Chrome browser).
That's why you have too many open connections indicated.
We have fixed this bug in the latest server build 2108.
It is under verification now.
3. We have added the new setting in server.properties
Code:
keep_alive.enabled=websocket,rtmp,rtmfp
This means you can enable / disable keep alives for a protocol.
For example you can enable keep alives for RTMP only:
Code:
keep_alive.enabled=rtmp
This setting was also introduced in 2108 and is under verification now.
 

Max

Administrator
Staff member
Each of such connections takes around of 30 k bytes, if it had publish/play in this connection.
One note. Even though it takes k bytes of memory, this memory is released after garbage collection.
So it seems it is not a huge issue.
Anyway it should be fixed in 2108.
 

phil

New Member
One note. Even though it takes k bytes of memory, this memory is released after garbage collection.
So it seems it is not a huge issue.
Anyway it should be fixed in 2108.
Hello Max, thanks a lot for your help.

One thing I'd like to ask is : What's the key difference between "keep_alive.enabled" and "keep_alive.algorithm"?
As I have pointed out earlier, I need to set keep_alive.algorithm in order to receive streams from FFMPEG. What is the best setting if this is a known issue?

Also, I have noticed that connections of "streams_websocket_out" are sometimes not released. For example,
upload_2017-3-8_23-0-37.png

Thanks,
 

Max

Administrator
Staff member
Hello
Code:
keep_alive.enabled
Here we list protocols where the keep alive is enabled.
Example:
Code:
keep_alive.enabled=rtmp,rtmfp
Keep alive will be enabled for rtmp and rtmfp protocols.
Code:
keep_alive.enabled=rtmp,websocket
You can also setup keep alive algorithm
NONE - do not use
INTERNAL - built in keep alive in the protocol using in-protocol bytes
HIGH_LEVEL - keep alive over the protocol using simple text messages ping and pong.
Applicable settings for keep_alive.algorithm:
keepalive.jpg

So you can disable keep alive for RTMP:
Code:
keep_alive.enabled=rtmfp,websockets
and switch all remaining used keep alives to INTERNAL
Code:
keep_alive.algorithm=INTERNAL
You can also set shorter keep alive intervals:
Code:
keep_alive.server_interval =2000
keep_alive.probes =2
In such case disconnect will be fired in 4 seconds (2x2) if connection was lost.
 
Top