com.flashphoner.server.G.A.E: No free ports available

A

AlanM

Member
I received this issue this morning during an audio-only stream. A few hundred subscribers were online at the time. Existing users were able to maintain their audio stream, but new subscriber connections were not allowed. All CPUs started going to ~100% usage. I tried stop/start on webcallserver service. That worked for a brief minute or so, then it stopped allowing new subscribers again.

I sent all logs in earlier and received a response of "We have fixed similar issue in latest builds." and was asked to duplicate the post here.

WCS version: 5.0.2712
 
A

AlanM

Member
Looking at previous forum posts:
During server operation, the number of media ports decreased
Port manager was updated in 2620 (We are on a newer version)
We have the default enable_extended_logging setting. Do we need to use "enable_extended_logging = false"?
No free media ports
Fixed in assembly 2601
===============================================================================

Do we need to increase our port_from and port_to range?
This is our current flashphoner.properties config:
ip =xxx.xxx.xxx.xxx
ip_local =xxx.xxx.xxx.xxx
port_from =30000
port_to =31000
media_port_from =31001
media_port_to =32000
waiting_answer =60
user_agent =Flashphoner/1.0
balance_header =balance
cost_header =cost
video_enabled =true
domain =
outbound_proxy =
outbound_port =
log_level =3
enable_context_logs =false
rtp_activity_detecting =true,60
sip_msg_listener =com.flashphoner.sdk.sip.ChangeCallIdListener
call_record_listener =com.flashphoner.server.client.DefaultCallRecordListener
dtmf =rfc2833
auto_login_url =/usr/local/FlashphonerWebCallServer/conf/account.xml
get_callee_url =/usr/local/FlashphonerWebCallServer/conf/callee.xml
codecs =opus,alaw,ulaw,g729,speex16,g722,mpeg4-generic,telephone-event,h264,vp8,flv,mpv
codecs_exclude_sip =mpeg4-generic,flv,mpv
codecs_exclude_streaming =flv,telephone-event
codecs_exclude_sip_rtmp =opus,g729,g722,mpeg4-generic,telephone-event,vp8,mpv
on_record_hook_script =on_record_hook.sh

rtmp_transponder_stream_name_prefix =rtmp_

client_mode=false

rtc_ice_add_local_component=true
 
Max

Max

Administrator
Staff member
Hello,

The issues from the other threads and some others were fixed before 2712.
Increasing port ranges can be tried.
Will inform when there is any update.
 
A

AlanM

Member
Any updates on this? We've tried expanding the port range up to as many at 16k ports. Even when he have just 4k ports, sometimes we get "no free ports" error and sometimes the CPUs go to very high usage (80-90%+, all cores). With the default 1k ports, it always stops when it hits the "no free ports" error. It seems like each subscriber client is using 4 ports?

Our next step is to disable extended logging.

Do you have a config guide for running a lean server without all the extra debugging/management overhead? The only functionality we need is:
WebRTC broadcast -> Multiple WebRTC subscribers (Audio only for some and audio/video combined for most events)
* We have used stream recording, but are willing to disable this if it means more stability
* We would like to run multiple events on a server, but we can run a single event per server if it means more stability
 
A

AlanM

Member
Just got results on our latest test with extended logging disabled. Changing that logging option did not help
 
Max

Max

Administrator
Staff member
Hello
Server refuses the sent private key.
Please check on your end.
 
Max

Max

Administrator
Staff member
We have successfully logged using default ec2-user login.
Logs seem clear excepting ports issue.

We have added this line to WCS_HOME/conf/log4j.properties
Code: 
log4j.logger.MediaPortManager=DEBUG
If this issue is reproduced again, please let us know, we will investigate logs.
Port tracer logs are available in WCS_HOME/logs/server_logs/flashphoner.log
So you can see ports utilization in realtime.
 
A

AlanM

Member
Load testing clients are still connected. We are receiving the error in our logs again:
com.flashphoner.server.G.A.E: No free ports available

Can you ssh in and review?
 
Max

Max

Administrator
Staff member
We have updated WCS version to 2748 and set additional setting in flashphoner.properties
Code: 
media_port_from=31001
media_port_to=50000
custom_ice_agent=true
Please check again.
 
Max

Max

Administrator
Staff member
You can also apply other listed tunings
flashphoner.properties
Code: 
rtp_receive_buffer_size=13107200
rtp_send_buffer_size =13107200
linux core
Code: 
ip link set txqueuelen 2000 dev eth0
 
A

AlanM

Member
We've made the additional config changes you listed. We were able to get to about 800 subscribers for an audio-only stream. Once it got closer to 900 subscribers, the audio started getting distorted.
Are there any additional options to squeeze out more performance? Can we turn off logging or any non-critical process management to reduce overhead?

Thanks
 
You must log in or register to reply here.
Top