RTSP port pool exhaustion ("No free ports available") - port leak on failed/dead camera reconnects

araiki

New Member
Subject: RTSP port pool exhaustion ("No free ports available") after upgrade 5.2.2192 → 5.3.414 - port leak on failed/dead camera reconnects

Environment

  • WCS version: upgraded from 5.2.2192 to 5.3.414 (Docker image flashphoner/webcallserver:5.3.414)
  • Deployment: single-instance Linux VM (Docker, --network=host), JVM -Xmx8g -Xms8g, ZGC garbage collector enabled
  • Relevant config (flashphoner.properties):
    rtsp_port_from = 35001
    rtsp_port_to = 40000
    rtsp_interleaved_mode = true
    rtsp_activity_timer_timeout = 5000
    rtsp_ports_auditor_interval = 1000 # lowered from default 10000 during troubleshooting
    rtsp_server_enabled = false


  • Use case: server acts as RTSP client only, pulling video streams from ~hundreds of IP cameras (UDP transport) over site-to-site VPN links, and re-serving via WebRTC to browser clients. No RTSP/RTMP/HLS server-side features enabled.
Symptom The RTSP port pool (5000 configured ports, ~2500 usable concurrent sessions) fills up over several hours until fully exhausted, at which point every new RTSP stream startup fails for every camera (not just the ones with connectivity issues), with:

ERROR RtspAgents - Can't create rtspClient
com.flashphoner.server.B.A.G: No free ports available


This is a full production outage: no camera can be viewed until the WCS process is restarted.

Timeline
  • After the version upgrade + VM restart, the pool went from empty to fully exhausted in a couple of hours.
  • After a service restart clearing the pool, it exhausted again in ~4.5 hours
  • We lowered rtsp_ports_auditor_interval from the default 10000 to 1000 ms as a mitigation attempt. This reduced the net port-consumption rate by roughly 25% but did not stop the leak.
Metrics observed (via your Prometheus port_stats exporter)
  • ports_rtsp_busy climbs steadily and monotonically over hours; ports_rtsp_free drops to 0 at exhaustion.
  • Crucially, ports_rtsp_leaked and ports_rtsp_quarantine stay low and noisy throughout (single digits to ~50, no sustained upward trend) — they do not account for the missing capacity. This suggests affected ports never enter your leak/quarantine bookkeeping at all, they simply remain marked "busy" indefinitely.
  • Cross-referencing our own application logs: the rate of new stream-open requests we send to your REST API (rest-api/rtsp/startup) is roughly flat over many hours, while ports_rtsp_busy keeps net-increasing, implying that a meaningful fraction (~15-25%) of session teardown/reconnect cycles never release their previously allocated port back to the pool.
Suspected root cause We have confirmed several specific cameras in our fleet are permanently unreachable/dead (network/hardware issue on the camera side, outside our control), and they generate a disproportionate share of failed RTSP connection attempts (dozens of "Connection failed" retries per camera within a 3-hour window, versus single digits for the rest of the healthy fleet). We suspect rtsp_refresh_requests_limit (default: 5, "Stop sending refresh requests if the limit has been reached") only stops the keepalive probing on a dead/unresponsive session, without actually closing the session and returning its allocated port(s) to the free pool. Repeated reconnect attempts against these permanently-dead cameras would then each leak a port with no way to recover it short of a full WCS restart.

With RtspPortManager=DEBUG enabled, we observe the leaked/quarantine buckets cycling correctly and staying small (~10 ports) at steady state, while busy keeps growing over hours. This strongly suggests the leaked-port auditor only catches a specific subset of dead sessions (those with an explicit disconnect event), while sessions where the camera goes silent without a clean signal never enter the leaked/quarantine pipeline at all, and remain permanently counted as busy.

Questions for support
  1. Is there a known regression between 5.2.2192 and 5.3.414 in how RTSP client sessions release their allocated ports on connection failure / repeated failed keepalives?
  2. Does hitting rtsp_refresh_requests_limit actually close the RTSP session and free its port(s), or does it just stop the keepalive probing while leaving the port allocated?
  3. Is there a setting to force a hard session timeout/port release specifically for camera-side connection failures (as opposed to rtsp_activity_timer_timeout, which only covers "no subscriber" streams)?
 
Last edited:
Good day.
We reproduced the issue and raised the ticket WCS-5122. Will let you know about fix.
 
Top