Add delay to player

A

Azhar Ali

Member
Hi Max,

We still have the same problems of crackly sound on rtmp. It's pretty bad and we are getting a lot of complaints from our customers. We are a new service and worried people putting on forums and start complaining. We have gone from great voice quality to broken. Can you please help on this.

I have attached below the stats page when voice was not good but unfortunately, client_log_level=DEBUG was set as ERROR and config can't be changed on the edge without modifying the image and template of the server.

If I directly change the settings on the edge CDN server, it reverts back somehow, not sure why (even if the instance is not recreated in the instance group)

The load is not huge but as more and more people start to connect, it starts to get crackly. WebRTC is all good.

I will set that after the end of today and try and generate them tomorrow again but anything you can look at meanwhile will be a great help.

Code: 
-----Connection Stats-----
connections=521
connections_rtmfp=0
connections_websocket=521
connections_hls=0
-----Port Stats-----
ports_media_free=9489
ports_media_busy=506
ports_media_quarantine=4
ports_wcs_agents_free=996
ports_wcs_agents_busy=2
ports_wcs_agents_quarantine=0
-----Stream Stats-----
streams_webrtc_in=2
streams_webrtc_out=495
streams_websocket_out=1
streams_rtmfp_in=0
streams_rtmfp_out=0
streams_rtmp_in=0
streams_rtmp_out=0
streams_hls=0
streams_viewers=rtmp_FJSQUAWK/388;FJSQUAWK/108
streams_synchronization=rtmp_SQUAWK/0;FJSQUAWK/0
stats_size=0
streams_rtsp_in=0
streams_rtsp_out=0
streams_rtmp_client_out=0
streams_play_rate=1
streams_stop_rate=0
-----Native Resources-----
native_resources=9705600,RESAMPLER:48000/48000,0;140569288728960,opus,4190269440;9291328,mpeg4-generic,-2761046
native_resources.audio_codecs=2
native_resources.audio_resamplers=1
native_resources.video_transcoders=0
native_resources.video_decoders=0
native_resources.video_encoders=0
native_resources.writers=0
-----Core Stats-----
core_heap_memory_used=1914699776
core_java_committedMemory=17602019463168
core_java_threads=114
core_java_freePhysicalMemorySize=6735626240
core_java_arch=amd64
core_java_availableProcessors=4
core_java_freeSwapSpaceSize=0
core_java_maxFileDescriptorCount=20000
core_java_open_file_descriptors=2236
core_java_cpu_usage=12.18
core_java_totalPhysicalMemorySize=16655777792
core_java_totalSwapSpaceSize=0
core_java_uptime=22074665
core_java_version=12.0.2
-----Call Stats-----
sip_processed_calls=0
sip_calls_state=established/0;trying/0;ringing/0;ring/0;ring_media/0;hold/0;busy/0;finish/0;session_progress/0;pending/0;failed/0
sip_calls=0
sip_calls_established=0
sip_calls_in=0
sip_calls_out=0
sip_calls_per_second=0.00
-----Sip Stats-----
sip_registered=0
-----Recording Stats-----
recording_sessions=0
-----System Stats-----
system_java_cpu_usage=0.00
system_java_load_average=0.62
-----Network Stats (Mbit/s)-----
global_bandwidth_in=0.000
global_bandwidth_out=0.000
-----Version info-----
wcs_version=5.2.768-230beb3f2d8801d729bd8293e94ae6b7ad2bf65b
wcs_client_version=0.5.28.2753-8fcfd6021df00c6b5b6581ab61f0a27516bb5448
-----Errors info-----
javax.net.ssl.SSLException=2
java.io.IOException=1
com.flashphoner.server.commons.rmi.operations.exception.ClientNotFoundException=74
java.lang.IllegalArgumentException=2
java.lang.reflect.InvocationTargetException=1
-----Degraded streams-----
degraded_streams=
degraded_streams_percent=0
-----Transcoding info-----
transcoding_video_decoding_resolutions=0x0/1
transcoding_video_decoding_average_time=0x0/-1.0
transcoding_video_decoding_max_time=0x0/-1
transcoding_video_decoding_average_queue_size=0x0/0.0
transcoding_video_decoding_max_queue_size=0x0/0
transcoding_video_decoding_load=0
transcoding_video_encoding_resolutions=0x0/1
transcoding_video_encoding_average_time=0x0/0.0
transcoding_video_encoding_max_time=0x0/0
transcoding_video_encoding_average_queue_size=0x0/0.0
transcoding_video_encoding_max_queue_size=0x0/0
transcoding_video_encoding_load=0
-----CDN info-----
cdn_version=2.5
cdn_role=EDGE
cdn_group=
 
Max

Max

Administrator
Staff member
Azhar Ali said:
native_resources=9705600,RESAMPLER:48000/48000,0;140569288728960,opus,4190269440;9291328,mpeg4-generic,-2761046
Click to expand...
If this is Edge server statistics page, you have excessive AAC to Opus transcoding on Edge server. This should be excluded by adding the following parameter to flashphoner.properties on Edge
Code: 
codecs_exclude_cdn=mpeg4-generic,alaw,ulaw,g729,speex16,g722,telephone-event,flv
Azhar Ali said:
and config can't be changed on the edge without modifying the image and template of the server.
Click to expand...
Yes, this is known autoscaling issue.
You can set up a single Edge (without LB and autoscaling) for tests.
We also raised internal ticket WCS-2899 to try to reproduce the issue in our load tests.
 
A

Azhar Ali

Member
Hi Max,
There stats were from the edge server. Would you like to see them on the origin as well?

If this is Edge server statistics page, you have excessive AAC to Opus transcoding on Edge server. This should be excluded by adding the following parameter to flashphoner.properties on Edge
Click to expand...
That setting is already present in the settings on the edge server. Please see the contents of the file below

Code: 
# Config flashphoner.properties

#webrtc ports range
port_from = 20000
port_to=20002
media_port_from        =20002
media_port_to          =39999
#codecs
codecs                   =opus,alaw,ulaw,g729,speex16,g722,mpeg4-generic,telephone-event,h264,vp8,flv,mpv
codecs_exclude_sip       =mpeg4-generic,flv,mpv
codecs_exclude_streaming =flv,telephone-event
codecs_exclude_sip_rtmp  =opus,g729,g722,mpeg4-generic,vp8,mpv
codecs_exclude_cdn=mpeg4-generic,alaw,ulaw,g729,speex16,g722,telephone-event,flv
#websocket ports
ws.port                 =8080
wss.port                =8443
cdn_enabled=true
cdn_ip=xx.xxx.x.xx
cdn_role=edge
cdn_point_of_entry=xx.xxx.x.x
cdn_nodes_resolve_ip=false
rtmp_transponder_stream_name_prefix =rtmp_
rtmp_transponder_full_url=true
client_mode=false
rtc_ice_add_local_component=true
webrtc_cc_min_bitrate=2000000
webrtc_cc_max_bitrate=2000000
stream_record_policy=template
stream_record_policy_template={streamName}-{mediaSessionId}

rtmp.port               =1935
rtmfp.port                       =1935
#keep_alive_algorithm may be INTERNAL, NONE, HIGH_LEVEL
keep_alive.algorithm       =HIGH_LEVEL
keep_alive.peer_interval   =2000
keep_alive.server_interval =5000
keep_alive.probes          =10
keep_alive.enabled = websocket,rtmfp
#Reliability: on, partial, off
video_reliable          =partial
audio_reliable          =partial
audio_frames_per_packet =6
burst_avoidance_count   =100
flush_audio_interval    =80
flush_video_interval    =0
disable_manager_rmi=false
enable_extended_logging=true
rest_max_connections=20
rtp_receive_buffer_size=131072
rtp_send_buffer_size =131072
rtp_activity_timeout=4320000
rtp_activity_video=false
webrtc_cc2_twcc=false
streaming_video_decoder_fast_start=false
client_log_level=DEBUG
enable_extended_logging=false
Normally, I understand and would not request this but is it possible to have a look at the servers between 1pm-3pm UTC time and see if we can get to the bottom of the issue? You have the ssh details in the previous emails.

Regards
Azhar
 
Max

Max

Administrator
Staff member
Azhar Ali said:
That setting is already present in the settings on the edge server. Please see the contents of the file below
Click to expand...
Ok, we see. Transcoding Opus -> AAC mentioned in this post occurs probably due to this
Azhar Ali said:
streams_websocket_out=1
Click to expand...
Seem like you have on MSE subscriber, in this case audio is sent to client using AAC codec.
Azhar Ali said:
Normally, I understand and would not request this but is it possible to have a look at the servers between 1pm-3pm UTC time and see if we can get to the bottom of the issue? You have the ssh details in the previous emails.
Click to expand...
We schedule this.
Also we'll try to reproduce the issue in load tests.
 
A

Azhar Ali

Member
that's interesting, if not support AAC solve this issue, for now, I am happy to go for that. If that means, it doesn't work in IE, thats fine as well.

If thats the temporary solution, how do we go about doing that?

thanks for taking the time to look at the server around that time.
 
Max

Max

Administrator
Staff member
Azhar Ali said:
thanks for taking the time to look at the server around that time.
Click to expand...
We collected some logs and now analyzing them
Azhar Ali said:
that's interesting, if not support AAC solve this issue, for now, I am happy to go for that. If that means, it doesn't work in IE, thats fine as well.
If thats the temporary solution, how do we go about doing that?
Click to expand...
The problem is probably not in audio transcoding.
The temporary solution can be to spread users by a number of servers, no more than 300 subscribers to RTMP stream for example. But it requires to deploy at lease 2 servers and do some job at backend to count subscribers and pass servers WSS endpoint to frontend.
 
Max

Max

Administrator
Staff member
We checked log collected. It seems like RTMP buffer issue that we will fix in ticket WCS-2899.
As workaround, we recommend:
-in Origin settings, set RTMP buffer polling interval to 20 milliseconds
Code: 
rtmp_out_buffer_polling_time=20
- in client application, enable TCP transport:
Code: 
session.createStream({
    name: streamName,
    display: remoteVideo,
    transport: "TCP"
}).on(STREAM_STATUS.PENDING, function (stream) {
...
}).play();
in Media Device example it can be enabled by switch "Transport"
1600927202766.png

If you make the changes above, we could look at your servers again between 1pm-3pm UTC and collect some additinal logs.
 
Max

Max

Administrator
Staff member
Azhar Ali said:
Would TCP effect the latency on webrtc stream? We cant afford to increase the latency for paid subscribers?
Click to expand...
For audio stream, the latency should not be perceptible. And this is just temporary solution until we fix RTMP buffer.
 
A

Azhar Ali

Member
Hi Max,

I have updated the client application to TCP and updated the origin settings as suggested. Should be good to collect the logs.

thanks for the help.

Regards
Azhar
 
Max

Max

Administrator
Staff member
Good day.
We working on RTMP buffer fix to pass packets more smoothly, let you know here when fix will be available.
 
Max

Max

Administrator
Staff member
Hello,

Checked the stream today, but the number of subscribers remained less than 490, and the audio quality issue has not been reproduced.
 
A

Azhar Ali

Member
Hi Max,

Thanks for checking, it seems to be intermittent. I can confirm, we didn't had any issues either yesterday.

Regards
Azhar
 
A

Azhar Ali

Member
Hi Max,

Is there any ETA on this? We have been getting regular complaints that WebRTC stream is delaying by 3-4 seconds. Do you think its down to TCP? We never had this issue before we made that change for RTMP stream to work.

Regards
Azhar
 
Max

Max

Administrator
Staff member
Good day.
Azhar Ali said:
We have been getting regular complaints that WebRTC stream is delaying by 3-4 seconds. Do you think its down to TCP?
Click to expand...
Yes, TCP transport may give such delay.
If switching to TCP helps to impove sound quality, it seems like networking issue specific for Google Cloud: media packets can be lost under load due to grouping them in buffer. TCP prevents packet losses, but adds delay waiting for packets will gather.
We recommend to enable TCP only for free subscribers, leaving UDP for paid ones. In this case, paid subscribers will receive a stream without delay (and there is no buffering for them), and free subscribers will receive a buffered delay plus TCP delay, but acceptable sound quality. We mentioned above how to enable TCP on per client basis.
Also, increase RTMP buffer polling frequency to 10 millisecond period
Code: 
rtmp_out_buffer_polling_time=10
This should also reduce packet losses and, therefore, reduce a delay.
 
Last edited:
You must log in or register to reply here.
Top