ICE Timeout on AWS Marketplace AMI

Hi,

I am trying to stream in AWS EC2 by using Market place AMI (https://aws.amazon.com/marketplace/pp/B07ZVRPH5G?qid=1599065528391&sr=0-1&ref_=srh_res_product_title). I can successfully publish stream but when I installed turnserver (coturn), I can't and it always fails with message "Failed by ICE Timeout"

I know it should be some ports block issue, but when I done the same WCS server setup (along with turnserver) in another EC2 by manually downloading the package it works perfectly.

The security groups and WCS build versions are same for both instances, but instance launched using marketplace AMI always fails with ICE timeout

1599065893172.png


I have sent an email with report logs of that instance, Please check
 

Max

Administrator
Staff member
Good day.
Unfortunately, we see the problem in logs, bu we cannot see the reason.
Please clarify the following:
- Does internal TURN server also not work? If yes, please check if TCP port 3478 is opened.
- Coturn server is installed on the same instance? If yes, please check if you test with relaying disabled
1599117194454.png

or, at code level, the parameter "iceTransportPolicy": "relay" is not used:
Code:
Flashphoner.createSession({urlServer: url, mediaOptions: {"iceServers": [ { 'url': 'turn:test.flashphoner.com:3478?transport=tcp', 'credential': 'coM77EMrV7Cwhyan', 'username': 'flashphoner' } ]}})
If this does not help, please provide us SSH access to the instance using this link.
 
Last edited:
- Does internal TURN server also not work? If yes, please check if TCP port 3478 is opened.
Yes, I am sure. TCP port 3478 is opened. But FYI, I am accessing the turnserver using load balancer from 443, where port 443 is pointed to 3479 which is the listening port of turnserver config file.


- Coturn server is installed on the same instance? If yes, please check if you test with relaying disabled
Yes, coturn is installed on the same server. I tried with disabling relay and it still not working.

FYI, the same thing is done on new instance where WCS is installed manually and it works well whereas it doen't with marketplace AMI.
 

Max

Administrator
Staff member
But FYI, I am accessing the turnserver using load balancer from 443, where port 443 is pointed to 3479 which is the listening port of turnserver config file.
This will not work properly in production. TURN server usage requires 2 ports to be forwarded simultaneously to the same instance behind the load balancer: WSS port (8443 for example) and TURN server port (3478 for example). But those are different entry points at load balancer point, so it can forward them to different instances.
Please try to deploy the separate TURN server which will not be behind load balancer.
 
I am using Classic Load Balancer where based on ports it will point to the respective instances behind it. FYI, this setup is currently in production well almost for 3 months and using the marketplace AMI breaks this.

I will share you the instance credentials in the mail so that you can look on that.
 
Hello Max,

I have sent an email with server credentials where instance is launched using Marketplace AMI.
Please share your IP so that I will open the SSH port.
Please check and let know if you have questions.
 

Max

Administrator
Staff member
Good day.
We successfully published stream using credentials you provided
1599732766674.png


1599732680340.png

Please note that we do not use relay option:
- Using this code, stream is published
Code:
Flashphoner.createSession({
    urlServer: 'wss://webrtc-dev.********.***:8443',
    mediaOptions: {
        "iceServers": [{
        'url': 'turns:webrtc-dev.********.***:443?transport=tcp',
        'credential': 'coM77EMrV7Cwhyan',
        'username': 'flashphoner'       
        }]
    }
})
- Using this code, stream publishing is failed by ICE timeout
Code:
Flashphoner.createSession({
    urlServer: 'wss://webrtc-dev.**********.***:8443',
    mediaOptions: {
        "iceServers": [{
        'url': 'turns:webrtc-dev.*********.***:443?transport=tcp',
        'credential': 'coM77EMrV7Cwhyan',
        'username': 'flashphoner'       
        }],
        "iceTransportPolicy": "relay"
    }
})
When iceTransportPolicy: relay option is used, TURN server allows WebRTC connection to be established directly between publishing client and server. But it cannot be established because load balancer does not forward WebRTC UDP ports.
When this option is not used, as recommenede in the post above, client establishes connection with TURN server only, using its TCP port (443 in this case). TURN server connects to WCS using WebRTC UDP ports, and wraps all media traffic to one TCP connection which load balancer can easily forward.
 
Great Max, Apart from "iceTransportPolicy": "relay" is there any other changes done on the server?

And one more question, I have another server with latest WCS build which is installed manually and using coturn and LB setup similar to the server I shared.
Even when "iceTransportPolicy": "relay" is enabled, I can able to publish stream there but why it is not working in the server I shared.
Just to know the working logic behind both servers.
 

Max

Administrator
Staff member
Apart from "iceTransportPolicy": "relay" is there any other changes done on the server?
No, we did not touch server setup.
And one more question, I have another server with latest WCS build which is installed manually and using coturn and LB setup similar to the server I shared.
Even when "iceTransportPolicy": "relay" is enabled, I can able to publish stream there but why it is not working in the server I shared.
We tested the case on our AWS instances: TURN server (internal and external on the same instance) works with relay policy disabled and does not work with relay policy enabled, both for manually deployed instance and AWS Marketplace one. Without load balancer, publishing and playing streams via TURN directly from server, we also got the same results.
So it seems like the issue may be in security group or iptables setup. This is even documented here.
 
@Max If I disable "iceTransportPolicy": "relay", turnserver username and password credentials are not getting validated.
I mean client can able to publish stream even if the username/password doesn't matches with the turnserver.

I think only if we enable "iceTransportPolicy", credentials will get authenticated and connects through turnserver to WCS.

FYI, UDP ports are open for the mediaport range.
 
Last edited:

Max

Administrator
Staff member
I think only if we enable "iceTransportPolicy", credentials will get authenticated and connects through turnserver to WCS.
Yes, this is TURN server default behaviour.
If you are using coturn on the same instance, please try to add the following parameter to flashphoner.properties
Code:
rtc_ice_add_local_interface=true
Then enable "iceTransportPolicy": "relay" and test. If this does not help, try to use external TURN server as workaround (not in the same instance).
 

Max

Administrator
Staff member
We've tested on the latest hourly AMI including 5.2.780 (and we strongly recommend update to this version if you're accessing WCS web interface from outside due to hot security fix) with coturn installed from epel-release
Code:
sudo amazon-linux-extras install epel
sudo yum install coturn
We've added the parameter to WCS configuration
Code:
rtc_ice_add_local_interface=true
We've configured coturn as follows (IP addresses are partially hidden)
Code:
# This post should be opened in security group and allowaed in instance iptables
listening-port=443
# Listen on internal IP only
listening-ip=172.31.xx.xxx
# Set up external IP option as coturn manual recommends
external-ip=54.75.x.xxx/172.31.xx.xxx
# Set user and password
user=username:password
# Set the domain
realm=yourdomain.com
# Disable UDP listener
no-udp
# SSL certificate and key
cert=/etc/pki/coturn/public/cert.pem
cert=/etc/pki/coturn/private/cert.key
Also, we've added the following parameter to [Service] section of /lib/systemd/system/coturn.service file:
Code:
...
[Service]
...
AmbientCapabilities=CAP_NET_BIND_SERVICE
[Install]
...
This is a workaround for coturn issue in AWS instance to bind port 443
Streaming is working with the setup above, all the traffic is relayed via coturn
1601608731475.png
 
Last edited:
@Max I updated the WCS build to latest version 5.2.780 and done the same configuration as said above except I am using a custom port 3479 as listening port for turnserver. But it is not working.

listening-port=3479
listening-ip=172.31.xx.xxx
external-ip=xx.xxx.xxx/172.31.xx.xxx
user=username: password
realm=mydomain.com
no-udp
This port (3479) is configured as target port and 443 is the source port in AWS Classic load balancer. And I think port 3478 is used by flashphoner.
Is this custom listening port causing any conflicts with the turnserver and flashpnoner?

Also I ensured this port (3479) is not used by any other process.

Also updated the coturn service file with
AmbientCapabilities=CAP_NET_BIND_SERVICE
 
Last edited:

Max

Administrator
Staff member
Is this custom listening port causing any conflicts with the turnserver and flashpnoner?
No. The port 443 is opened in AWS Marketplace instance by default, but 3479 is not. Please check iptables settings and add this port.
Note that adding the port to security group does not open it in instance iptables.
You can also flush instance iptables by iptables -F, because it may be excessive if security group is set correctly.
 
Max, the ports are already open but server with marketplace AMI still not working.
As said before in this post, server with manual WCS installation works whereas server launched with Marketplace AMI not working.

I have sent an email with credentials of both servers. Can you please check what is missing with the server that is launched using marketplace AMI?

Thank you.
 

Max

Administrator
Staff member
We've checked your setup.
First, in Marketplace AMI, you missed iptables entry for TCP port 3479 as we mentioned above. You can just flush iptables with iptables -F command.
We also recommend to set listening-ip and external-ip coturn parameters as described above.
In this case, direct publishing to the server via coturn works
1602215155503.png


But we noticed that publishing via coturn does not work using LB entry point, both for marketplace AMI and custom AMI
1602215299155.png





Traffic dump shows no incoming packets to coturn port 3479, but to WCS websocket port 8080 only. Looks like LB does not redirect packets to instance port 3479.
So you should:
1) Finish marketplace AMI setup
2) Check load balancer setup.
Please also note that if there are more than one instance behind the load balancer, it may probably pass websocket connection to one instnce and coturn connection to another. There is no way to redirect two different load balancer ports to one instance.
We recommend you to set up one dedicated coturn server instance to use with all the instances behind the load balancer.
 
Top