Getting StreatStatus.Failed when 200+ viewers connect to stream

A

Azhar Ali

Member
Hi,
Is there anyway to check what could be reason for getting StreamStatus.Failed when a viewer tries to connect to the stream. This only happens when there are several users connect to it. If you retry it sometimes work and sometimes returns failed. There is no message passed back for the reason for it.

Server is c4.8xlarge and one have one stream and cpu was only 3%
 
Max

Max

Administrator
Staff member
You have not enough configured ports to perform load tests.
Try to configure flashphoner.properties
Here we leave just two SIP ports to release ports capacity for streaming
Code: 
port_from=20000
port_to=20002
Here we set extended port range to serve 200 and more viewers
Code: 
media_port_from=20003
media_port_to=39999
Make sure all these ports are open on firewall.
 
A

Azhar Ali

Member
Hi,
I have made those changes and added the ports onto the inbound rules for the AWS. See attached image

Now I can't stream at all. After calling the createStream function I don't get any callbacks for publishing. I have also tested this in the demo app on our server and its also not working.

I have emailed the logs again.
 
Max

Max

Administrator
Staff member
Hello.
Please check firewall rules on your server. iptables -L shows the following:
Code: 
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpts:30000:33000
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpts:30000:33000
So you should set up iptables to accept port range 20003:39999
 
A

Azhar Ali

Member
Thanks Max, that has helped and fixed the connection issue. I did the test again and this time managed to connect 500 real users onto the stream but the delay was over 20 seconds. It would start with 4-5 seconds and slowly would go over 20 seconds. We want to stream sub second for our users. When there is only few users its fine and you get sub second latency.

I have emailed the link of the video recording to show the delay and image of the stats page.

Any ideas?
 
Max

Max

Administrator
Staff member
Hello.
You are publishing FullHD stream while sharing 1080p screen in fact. So, try to limit publishing bitrate as described here to optimize channel load, it may help to reduce the latency.
 
A

Azhar Ali

Member
Hi Max,

Thank you for the suggestions. I have tried and applied them but that did not seem to have resolved the latency. I changed the constraints to below, in our use case we mostly need to share screen and reducing the frame rate is acceptable. Doing that has brought the latency down to 2 second or but viewer video freezes for split second and jumps forward, its like data lose or something. I have attached a video to show it. If you see at
Code: 
var constraints = {
        video: {
            width: 1920,            height: 1080,
            frameRate: { max: 7 }
            , minBitrate: 500            ,maxBitrate: 1000
        }
    };
Also attached the flashphoner.properties and wcs-core.properties which I have also changed after looking the document.
Emailed the link for the video and files.
 
Max

Max

Administrator
Staff member
Hello
We have checked logs.
Server's performance looks good and tuned properly.
We also noticed by IP that your instance is located in United States Virginia region.
It is default AWS region. However this might be a network problem if your location is not US or if location of 500 viewers is not US.
Try to relocate AWS instance to a region where you physically located and where users physically located. The streaming requires low ping (latency) and server close to end-user's region.
 
A

Azhar Ali

Member
Hi Max,

Thanks for looking. Most of our viewers and publishers will be in US but we in UK. The test i did and sent in the email was myself publishing from UK but a viewing on a separate VM on azure in US. Is there any reason why it freezes for half a second or so and then skips as I mentioned in the email.

Also do you think any implication in using the following constraints for HD streaming.
Code: 
var constraints = {
        video: {
            width: 1920,            height: 1080,
            frameRate: { max: 1 }
            , minBitrate: 500            ,maxBitrate: 500
        }
    };
Even with such low bitrate it can stream in HD which i don't understand how its possible.
 
Max

Max

Administrator
Staff member
Even with such low bitrate it can stream in HD which i don't understand how its possible.
Click to expand...
It is because you pass frameRate 1.
Full HD with 1 FPS has low bitrate.
Please check this manual related bitrate management
https://docs.flashphoner.com/displa...ement+when+capturing+WebRTC+stream+in+browser
As you can see you can set constant bitrate using SDP hooks only and for Chrome only:
https://docs.flashphoner.com/displa...streaminbrowser-Howtoenforcebitrateincreasing
Is there any reason why it freezes for half a second or so and then skips
Click to expand...
Sorry could you provide a link for freezing video. We received two videos but there are no freezes there.
The test i did and sent in the email was myself publishing from UK but a viewing on a separate VM on azure in US.
Click to expand...
This is a bad test if you record final video from your local screen because Azure Remote Desktop may add a dozen seconds of latency or lost frames.
Indeed you can't estimate quality of video stream remotely in real time because any RDP-like tool will cause huge latency and frame lost for video. Only way to check video quality is recording screen on the remote Azure Machine.

Recommendations:
1. Try constant bitrates.
Make sure your bitrate is constant in chrome://webrtc-internals charts.
2. Try to test either recording screen on the remote machine or test in the UK region.
3. You can just have a bad internet channel towards N. Virginia US AWS server. That's why you may see packet lost and freezes in your stream.
Try to relocate your instance to Europe London or Frankfurt.
 
A

Azhar Ali

Member
Hi Max
,
Thanks for the detailed answer. I will look into your suggestion and respond.
This is a bad test if you record final video from your local screen because Azure Remote Desktop may add a dozen seconds of latency or lost frames.
Click to expand...
My bad on it but me and my partner were also testing this from our homes and tiny freezing was happening e.g seconds number would either move to next second too quickly or have a tiny freeze which is ok when you are looking in the video but when you are talking at the same time you get small pauses.

Based on your experience, which data center would be idea for Users in US. If you know.
 
Last edited:
Max

Max

Administrator
Staff member
Hello.
We've checked your recording. It seems like packets are lost periodically. This confirms recommendation to either relocate your instance closer to publisher or to publish stream from US to US.
You also can use WebRTC over TCP to publish/play a stream, it may help to escape packet loss but can increase the whole latency from publisher to player. To use TCP set "transport" stream option, for example:
Code: 
    session.createStream({
        name: streamName,
        display: localVideo,
        cacheLocalResources: true,
        receiveVideo: false,
        receiveAudio: false,
        transport: "TCP"
    }).on(STREAM_STATUS.PUBLISHING, function (stream) {
    ...
    }).publish();
 
A

Azhar Ali

Member
Hi Max,

I have been trying to work this out. I am getting random freezes like i sent the video and checking chrome://webrtc-internals to see if there is any packet loss on either publisher or viewer side but the graph does not show any lost packets. Is there a way to check if there is def. packet loss or something else might be causing the issue.

I also tried to use the TCP as you suggested to check what king of latency there is, it would not even publish anything else which needs to be changed on the server to enable that?
 
Max

Max

Administrator
Staff member
Hello
it would not even publish anything
Click to expand...
Please check if TCP media ports in range media_port_from:media_port_to is opened in ASW instance security rules.
 
You must log in or register to reply here.
Top