Hello,
We are using WCS and WebRTC in production.
Every 8-9 days, Flashphoner freeze and refuse new connections.
In the log, we are seeing a LOT of :
I attached our Grafana Dashboards, we are not seeing saturation on media or wcs_agent ports during the incident (20H-21H40 on the graph).
We are using an old version (5.2.1794) with the configuration below.
We are going to upgrade to the latest version and increase the media_port range to 30000-32500.
We looked at this issue https://forum.flashphoner.com/threads/com-flashphoner-server-g-a-e-no-free-ports-available.11295/ but we don't have this much usage. Only 10-30 users and 100 streams maximum.
Are you aware of a bug in this version? (we found no trace of a fix in the release note)
Do you think it could be a configuration issue or something else? Thanks !
I did a netstat -laputen on the virtual machine during the incident, and there was only a few ESTABLISHED and TIME_WAIT connections. As you can see on the dashboard, metrics show a lot of available ports. Incident occurred two time in a row already. Very strange.
We are using WCS and WebRTC in production.
Every 8-9 days, Flashphoner freeze and refuse new connections.
In the log, we are seeing a LOT of :
Code:
20:35:50,987 ERROR RtspAgents - Can't create rtspClient
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: com.flashphoner.server.B.A.G: No free ports available
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at com.flashphoner.server.B.A.C.E(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at com.flashphoner.server.remote.E.D.allocatePort(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at com.flashphoner.server.remote.B.<init>(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at com.flashphoner.server.remote.E.D.<init>(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at com.flashphoner.server.remote.E.A.C(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at com.flashphoner.server.remote.E.A.createAgent(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at com.flashphoner.server.remote.E.A(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at com.flashphoner.server.remote.E.B(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at com.flashphoner.server.remote.L.A(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at com.flashphoner.server.remote.L.A(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at com.flashphoner.rest.server.NodeApi.startupRtspAgent(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at com.flashphoner.rest.server.rest_v2.RestRtspController.startup(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at jdk.internal.reflect.GeneratedMethodAccessor115.invoke(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at java.base/java.lang.reflect.Method.invoke(Method.java:567)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at com.flashphoner.rest.server.RestApiRouter.getResponse(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at com.flashphoner.rest.server.RestApiRouter.processRequest(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at com.flashphoner.server.http.handlers.RestApiRequestHandler.process(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at com.flashphoner.server.http.G.messageReceived(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at org.jboss.netty.channel.socket.nio.NioWorker.read(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at org.jboss.netty.channel.socket.nio.DeadlockAwareNioWorker.run(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at org.jboss.netty.util.ThreadRenamingRunnable.run(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(Unknown Source)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Mar 13 20:35:50 flashphonerprod000000 bash[3564236]: at java.base/java.lang.Thread.run(Thread.java:835)We are using an old version (5.2.1794) with the configuration below.
We are going to upgrade to the latest version and increase the media_port range to 30000-32500.
We looked at this issue https://forum.flashphoner.com/threads/com-flashphoner-server-g-a-e-no-free-ports-available.11295/ but we don't have this much usage. Only 10-30 users and 100 streams maximum.
Are you aware of a bug in this version? (we found no trace of a fix in the release note)
Do you think it could be a configuration issue or something else? Thanks !
I did a netstat -laputen on the virtual machine during the incident, and there was only a few ESTABLISHED and TIME_WAIT connections. As you can see on the dashboard, metrics show a lot of available ports. Incident occurred two time in a row already. Very strange.
Code:
ip = {{EXTERNAL_IP}}
ip_local = {{LOCAL_IP}}
# RTP/SRTP port range
media_port_from = 30000
media_port_to = 31000
# RTSP port range
rtsp_port_from = 35001
rtsp_port_to = 40000
# WSS port range
wcs_agent_port_from = 40001
wcs_agent_port_to = 45000
# SIP port range
port_from = 30000
port_to = 31000
rtsp_interleaved_mode = true
# codecs = h264 # Breaks audio sdp
force_client_requested_video_resolution = false
# If true, activate loadbalancer:
# load_balancing_enabled = true
# Suppress audio globally (not available in trial version)
suppress_audio = true
http_enable_paths = rest,action,shared,embed_player,empty,health-check
# Close inactive RTSP->WCS stream with no subscriber
rtsp_activity_timer_timeout = 30000
# Websockets port
ws.port = {{WS_PORT}}
wss.port = {{WSS_PORT}}
# Logs
client_log_level = ERROR
client_dump_level = 0
enable_extended_logging = false
# Disable all unused features
cli_enabled = false
https_server_enabled = false
manager_http_ports_enabled = false
# Disable WebRTC Selective Forwarding Unit
# This is only usefu when :
# - publishing a stream in a number of encodings (for example, 720p, 360p, 180p) with quality switching on the fly
# - building chat rooms
sfu_bridge_enabled = false
# Disable unused protocols on the client side
# We only expose RTSP stream with MSE over websocket or WebRTC
rtsp_server_enabled = false
rtmp_server_enabled = false
rtmfp_server_enabled = false
hls_server_enabled = false
# Disabled unused protocols on the proxy side (DVR and cameras in parkings)
hls_enabled = false