ICE Timeout on AWS Marketplace AMI

[manikandan]

Hi,

I am trying to stream in AWS EC2 by using Market place AMI (https://aws.amazon.com/marketplace/pp/B07ZVRPH5G?qid=1599065528391&sr=0-1&ref_=srh_res_product_title). I can successfully publish stream but when I installed turnserver (coturn), I can't and it always fails with message "Failed by ICE Timeout"

I know it should be some ports block issue, but when I done the same WCS server setup (along with turnserver) in another EC2 by manually downloading the package it works perfectly.

The security groups and WCS build versions are same for both instances, but instance launched using marketplace AMI always fails with ICE timeout

I have sent an email with report logs of that instance, Please check

 

[Max]

Good day.
Unfortunately, we see the problem in logs, bu we cannot see the reason.
Please clarify the following:
- Does internal TURN server also not work? If yes, please check if TCP port 3478 is opened.
- Coturn server is installed on the same instance? If yes, please check if you test with relaying disabled

or, at code level, the parameter "iceTransportPolicy": "relay" is not used:

Flashphoner.createSession({urlServer: url, mediaOptions: {"iceServers": [ { 'url': 'turn:test.flashphoner.com:3478?transport=tcp', 'credential': 'coM77EMrV7Cwhyan', 'username': 'flashphoner' } ]}})

If this does not help, please provide us SSH access to the instance using this link.

 

[manikandan]

- Does internal TURN server also not work? If yes, please check if TCP port 3478 is opened.

Yes, I am sure. TCP port 3478 is opened. But FYI, I am accessing the turnserver using load balancer from 443, where port 443 is pointed to 3479 which is the listening port of turnserver config file.

- Coturn server is installed on the same instance? If yes, please check if you test with relaying disabled

Yes, coturn is installed on the same server. I tried with disabling relay and it still not working.

FYI, the same thing is done on new instance where WCS is installed manually and it works well whereas it doen't with marketplace AMI.

 

[Max]

But FYI, I am accessing the turnserver using load balancer from 443, where port 443 is pointed to 3479 which is the listening port of turnserver config file.

This will not work properly in production. TURN server usage requires 2 ports to be forwarded simultaneously to the same instance behind the load balancer: WSS port (8443 for example) and TURN server port (3478 for example). But those are different entry points at load balancer point, so it can forward them to different instances.
Please try to deploy the separate TURN server which will not be behind load balancer.

 

[manikandan]

I am using Classic Load Balancer where based on ports it will point to the respective instances behind it. FYI, this setup is currently in production well almost for 3 months and using the marketplace AMI breaks this.

I will share you the instance credentials in the mail so that you can look on that.

 

[Max]

I will share you the instance credentials in the mail so that you can look on that.

Ok, we will check.

 

[manikandan]

Hello Max,

I have sent an email with server credentials where instance is launched using Marketplace AMI.
Please share your IP so that I will open the SSH port.
Please check and let know if you have questions.

 

[Max]

Good day.
We successfully published stream using credentials you provided

Please note that we do not use relay option:
- Using this code, stream is published

Flashphoner.createSession({
    urlServer: 'wss://webrtc-dev.********.***:8443',
    mediaOptions: {
        "iceServers": [{
        'url': 'turns:webrtc-dev.********.***:443?transport=tcp',
        'credential': 'coM77EMrV7Cwhyan',
        'username': 'flashphoner'       
        }]
    }
})

- Using this code, stream publishing is failed by ICE timeout

Flashphoner.createSession({
    urlServer: 'wss://webrtc-dev.**********.***:8443',
    mediaOptions: {
        "iceServers": [{
        'url': 'turns:webrtc-dev.*********.***:443?transport=tcp',
        'credential': 'coM77EMrV7Cwhyan',
        'username': 'flashphoner'       
        }],
        "iceTransportPolicy": "relay"
    }
})

When iceTransportPolicy: relay option is used, TURN server allows WebRTC connection to be established directly between publishing client and server. But it cannot be established because load balancer does not forward WebRTC UDP ports.
When this option is not used, as recommenede in the post above, client establishes connection with TURN server only, using its TCP port (443 in this case). TURN server connects to WCS using WebRTC UDP ports, and wraps all media traffic to one TCP connection which load balancer can easily forward.

 

[manikandan]

Great Max, Apart from "iceTransportPolicy": "relay" is there any other changes done on the server?

And one more question, I have another server with latest WCS build which is installed manually and using coturn and LB setup similar to the server I shared.
Even when "iceTransportPolicy": "relay" is enabled, I can able to publish stream there but why it is not working in the server I shared.
Just to know the working logic behind both servers.

 

[Max]

Apart from "iceTransportPolicy": "relay" is there any other changes done on the server?

No, we did not touch server setup.

And one more question, I have another server with latest WCS build which is installed manually and using coturn and LB setup similar to the server I shared.
Even when "iceTransportPolicy": "relay" is enabled, I can able to publish stream there but why it is not working in the server I shared.

We tested the case on our AWS instances: TURN server (internal and external on the same instance) works with relay policy disabled and does not work with relay policy enabled, both for manually deployed instance and AWS Marketplace one. Without load balancer, publishing and playing streams via TURN directly from server, we also got the same results.
So it seems like the issue may be in security group or iptables setup. This is even documented here.

 

[manikandan]

@Max If I disable "iceTransportPolicy": "relay", turnserver username and password credentials are not getting validated.
I mean client can able to publish stream even if the username/password doesn't matches with the turnserver.

I think only if we enable "iceTransportPolicy", credentials will get authenticated and connects through turnserver to WCS.

FYI, UDP ports are open for the mediaport range.

 

[Max]

I think only if we enable "iceTransportPolicy", credentials will get authenticated and connects through turnserver to WCS.

Yes, this is TURN server default behaviour.
If you are using coturn on the same instance, please try to add the following parameter to flashphoner.properties

rtc_ice_add_local_interface=true

Then enable "iceTransportPolicy": "relay" and test. If this does not help, try to use external TURN server as workaround (not in the same instance).

 

[Max]

We've tested on the latest hourly AMI including 5.2.780 (and we strongly recommend update to this version if you're accessing WCS web interface from outside due to hot security fix) with coturn installed from epel-release

sudo amazon-linux-extras install epel
sudo yum install coturn

We've added the parameter to WCS configuration

rtc_ice_add_local_interface=true

We've configured coturn as follows (IP addresses are partially hidden)

# This post should be opened in security group and allowaed in instance iptables
listening-port=443
# Listen on internal IP only
listening-ip=172.31.xx.xxx
# Set up external IP option as coturn manual recommends
external-ip=54.75.x.xxx/172.31.xx.xxx
# Set user and password
user=username:password
# Set the domain
realm=yourdomain.com
# Disable UDP listener
no-udp
# SSL certificate and key
cert=/etc/pki/coturn/public/cert.pem
cert=/etc/pki/coturn/private/cert.key

Also, we've added the following parameter to [Service] section of /lib/systemd/system/coturn.service file:

...
[Service]
...
AmbientCapabilities=CAP_NET_BIND_SERVICE
[Install]
...

This is a workaround for coturn issue in AWS instance to bind port 443
Streaming is working with the setup above, all the traffic is relayed via coturn

 

[manikandan]

Thank you Max. Will check and update.

 

[manikandan]

@Max I updated the WCS build to latest version 5.2.780 and done the same configuration as said above except I am using a custom port 3479 as listening port for turnserver. But it is not working.

listening-port=3479
listening-ip=172.31.xx.xxx
external-ip=xx.xxx.xxx/172.31.xx.xxx
user=username: password
realm=mydomain.com
no-udp

This port (3479) is configured as target port and 443 is the source port in AWS Classic load balancer. And I think port 3478 is used by flashphoner.
Is this custom listening port causing any conflicts with the turnserver and flashpnoner?

Also I ensured this port (3479) is not used by any other process.

Also updated the coturn service file with

AmbientCapabilities=CAP_NET_BIND_SERVICE

 

[Max]

Is this custom listening port causing any conflicts with the turnserver and flashpnoner?

No. The port 443 is opened in AWS Marketplace instance by default, but 3479 is not. Please check iptables settings and add this port.
Note that adding the port to security group does not open it in instance iptables.
You can also flush instance iptables by iptables -F, because it may be excessive if security group is set correctly.

 

[manikandan]

Max, the ports are already open but server with marketplace AMI still not working.
As said before in this post, server with manual WCS installation works whereas server launched with Marketplace AMI not working.

I have sent an email with credentials of both servers. Can you please check what is missing with the server that is launched using marketplace AMI?

Thank you.

 

[Max]

We've checked your setup.
First, in Marketplace AMI, you missed iptables entry for TCP port 3479 as we mentioned above. You can just flush iptables with iptables -F command.
We also recommend to set listening-ip and external-ip coturn parameters as described above.
In this case, direct publishing to the server via coturn works

But we noticed that publishing via coturn does not work using LB entry point, both for marketplace AMI and custom AMI

Traffic dump shows no incoming packets to coturn port 3479, but to WCS websocket port 8080 only. Looks like LB does not redirect packets to instance port 3479.
So you should:
1) Finish marketplace AMI setup
2) Check load balancer setup.
Please also note that if there are more than one instance behind the load balancer, it may probably pass websocket connection to one instnce and coturn connection to another. There is no way to redirect two different load balancer ports to one instance.
We recommend you to set up one dedicated coturn server instance to use with all the instances behind the load balancer.