araiki
New Member
Subject: RTSP port pool exhaustion ("No free ports available") after upgrade 5.2.2192 → 5.3.414 - port leak on failed/dead camera reconnects
Environment
ERROR RtspAgents - Can't create rtspClient
com.flashphoner.server.B.A.G: No free ports available
This is a full production outage: no camera can be viewed until the WCS process is restarted.
Timeline
With RtspPortManager=DEBUG enabled, we observe the leaked/quarantine buckets cycling correctly and staying small (~10 ports) at steady state, while busy keeps growing over hours. This strongly suggests the leaked-port auditor only catches a specific subset of dead sessions (those with an explicit disconnect event), while sessions where the camera goes silent without a clean signal never enter the leaked/quarantine pipeline at all, and remain permanently counted as busy.
Questions for support
Environment
- WCS version: upgraded from 5.2.2192 to 5.3.414 (Docker image flashphoner/webcallserver:5.3.414)
- Deployment: single-instance Linux VM (Docker, --network=host), JVM -Xmx8g -Xms8g, ZGC garbage collector enabled
- Relevant config (flashphoner.properties):
rtsp_port_from = 35001
rtsp_port_to = 40000
rtsp_interleaved_mode = true
rtsp_activity_timer_timeout = 5000
rtsp_ports_auditor_interval = 1000 # lowered from default 10000 during troubleshooting
rtsp_server_enabled = false
- Use case: server acts as RTSP client only, pulling video streams from ~hundreds of IP cameras (UDP transport) over site-to-site VPN links, and re-serving via WebRTC to browser clients. No RTSP/RTMP/HLS server-side features enabled.
ERROR RtspAgents - Can't create rtspClient
com.flashphoner.server.B.A.G: No free ports available
This is a full production outage: no camera can be viewed until the WCS process is restarted.
Timeline
- After the version upgrade + VM restart, the pool went from empty to fully exhausted in a couple of hours.
- After a service restart clearing the pool, it exhausted again in ~4.5 hours
- We lowered rtsp_ports_auditor_interval from the default 10000 to 1000 ms as a mitigation attempt. This reduced the net port-consumption rate by roughly 25% but did not stop the leak.
- ports_rtsp_busy climbs steadily and monotonically over hours; ports_rtsp_free drops to 0 at exhaustion.
- Crucially, ports_rtsp_leaked and ports_rtsp_quarantine stay low and noisy throughout (single digits to ~50, no sustained upward trend) — they do not account for the missing capacity. This suggests affected ports never enter your leak/quarantine bookkeeping at all, they simply remain marked "busy" indefinitely.
- Cross-referencing our own application logs: the rate of new stream-open requests we send to your REST API (rest-api/rtsp/startup) is roughly flat over many hours, while ports_rtsp_busy keeps net-increasing, implying that a meaningful fraction (~15-25%) of session teardown/reconnect cycles never release their previously allocated port back to the pool.
With RtspPortManager=DEBUG enabled, we observe the leaked/quarantine buckets cycling correctly and staying small (~10 ports) at steady state, while busy keeps growing over hours. This strongly suggests the leaked-port auditor only catches a specific subset of dead sessions (those with an explicit disconnect event), while sessions where the camera goes silent without a clean signal never enter the leaked/quarantine pipeline at all, and remain permanently counted as busy.
Questions for support
- Is there a known regression between 5.2.2192 and 5.3.414 in how RTSP client sessions release their allocated ports on connection failure / repeated failed keepalives?
- Does hitting rtsp_refresh_requests_limit actually close the RTSP session and free its port(s), or does it just stop the keepalive probing while leaving the port allocated?
- Is there a setting to force a hard session timeout/port release specifically for camera-side connection failures (as opposed to rtsp_activity_timer_timeout, which only covers "no subscriber" streams)?
Last edited: