AMI /etc/hosts, hostname

[sangsoo]

Hello. I have a question.
I am designing a service using "CDN2.0 + AWS Auto Scale setup".
The first time you start a WCS in Amazon EC2 Instance, there is only a localhost value in /etc/hosts (we added hostname during the test).
(only) 127.0.0.1 localhost localhost.localdomain

If no hostname exists, WCS starts, but the CDN node configuration fails, and RTMP publishing seems to fail at origin.
Should I add a private ip and hostname as a separate script every time an AMI instance increase? How should I manage this value?
Or is this design wrong?
And, Can I configure Auto Scale Setup for the WCS Origin server?

Best Regards.

 

[sangsoo]

After setting the flashphoner.properties of the edge server (below), cdn and rtmp injest also work properly.
cdn_ip = 0.0.0.0
cnd_nodes_resolve_ip = false

* Origin-edge aws az is different in test environment

 

[Max]

Good day.
We raised internal ticket (WCS-2366) and let you know results in this topic. Later we will tell you, how to configure Amazon EC2 Auto Scaling for Edge servers.

 

[Max]

Now we are still solving this problem and offer you workaround. To set local IP to cdn_ip setting while first start in AWS EC2 instance modify file /usr/local/FlashphonerWebCallServer/bin/webcallserver (diff is attached below):

    if [[ $PUBLIC_IP_STATUS -eq 0 && $LOCAL_IP_STATUS -eq 0 ]]; then
-    sed -i -e "s/^\(ip .*=\).*\$/\1$PUBLIC_IP/" $WCS_CONFIG
-    sed -i -e "s/^\(ip_local.*=\).*\$/\1$LOCAL_IP/" $WCS_CONFIG
+    sed -i -e "s/^\(ip[ \t]*=\).*\$/\1$PUBLIC_IP/" $WCS_CONFIG
+    sed -i -e "s/^\(ip_local[ \t]*=\).*\$/\1$LOCAL_IP/" $WCS_CONFIG
+    sed -i -e "s/^\(cdn_ip[ \t]*=\).*\$/\1$LOCAL_IP/" $WCS_CONFIG
    else
-    sed -i -e "s/^\(ip.*=\).*\$/\1/" $WCS_CONFIG
+    sed -i -e "s/^\(ip[ \t]*=\|ip_local[ \t]*=\|cdn_ip[ \t]*=\).*\$/\1/" $WCS_CONFIG
    fi
   fi

 

[Max]

Good day.
In WCS version 5.2.415 (and later), we fixed this problem. While first start AWS EC2 instance, sets cdn_ip from setting local IP.

 

[sangsoo]

Thank you for troubleshooting (5.2.415).
I will try to configure auto scaling for the edge server.

Best Regards.

 

[ett]

Hello, Max. Please help as the CDN at AWS.

Good day.
In WCS version 5.2.415 (and later), we fixed this problem. While first start AWS EC2 instance, sets cdn_ip from setting local IP.

Why is cdn_ip force rewritten to the local IP?
I used cdn_ip = hostname as you say in other threads,
As a result, cdn bind to $LOCAL_IP:8084

tcp        0      0 $LOCAL_IP:8084          0.0.0.0:*                   LISTEN

Of course, it cannot connect from external.

09:41:20,816 ERROR         CDNNodes - Thread-840 Failed to connect to node $PUBLIC_IP
09:41:20,816 INFO             C - CDNOutbound-BOSS-pool-34-thread-1 Exception on channel [id: 0x5e691fc7], No route to host

Perhaps you tell me to write hold_ip_settings = true?
Or is it correct to put the origin server's local IP in cdn_point_of_entry = ${origin local IP}?

Regards

 

[Max]

Hello,

is it correct to put the origin server's local IP in cdn_point_of_entry = ${origin local IP}?

Yes.

CDN settings for AWS instances:

- Origin

cdn_enabled=true
cdn_ip=<Origin local IP>
cdn_role=origin

- Edge

cdn_enabled=true
cdn_ip=<Edge local IP>
cdn_point_of_entry=<Origin local IP>
cdn_role=edge

 

[ett]

Are you assuming no external connection for CDN?
For example, [AWS Origin] <=> [GCP Edge], and [AWS us-west-1 Origin] <=> [AWS us-east-1 Edge]?
<https://forum.flashphoner.com/threads/cdn-edge-origins-distribution-question.11750/#post-18327>


In the documentation origin.flashponer.com, edge.flashponer.com it looks like a public IP rather than a local IP.
And you said

In case cdn_ip=<hostname>, the other nodes will be able to connect if the hostname resolves to the server public IP.

<https://forum.flashphoner.com/threads/cdn-edge-cant-connect-to-origin.11990/>
But cdn_ip never resolves to a public IP by sed -i -e "s/^\(cdn_ip[ \t]*=\).*\$/\1$LOCAL_IP/" $WCS_CONFIG


1. As far as I read these, I thought the CDN is able to be configured through the Internet not a intranet. Is this my mistake?

2. Is there a problem if I use hold_ip_settings = true to configure a CDN over the Internet?

Best regards

 

[Max]

Good day.

1. As far as I read these, I thought the CDN is able to be configured through the Internet not a intranet. Is this my mistake?

You can configure Origin as follows:

ip=<external ip>
ip_local=<local ip>
...
cdn_enabled=true
cdn_ip=<local ip>
cdn_role=origin
...

In this case, Origin will accept publishing client connections on external (public) IP, but CDN connections on local (internal) IP.
Edge server can be configured like this

ip=<external ip>
ip_local=<local ip>
...
cdn_enabled=true
cdn_ip=<local ip>
cdn_role=edge
cdn_point_of_entry=<origin local ip>
...

In this case, Edge will accept subscribers on public IP and will pull streams from Origin via private IP. You should set IP addresses, not hostnames.
As a result, CDN will be available for external clients, and private network will be used for internal CDN traffic.

For example, [AWS Origin] <=> [GCP Edge], and [AWS us-west-1 Origin] <=> [AWS us-east-1 Edge]?

Unfortunately, private IP addresses can be in different subnets for different datacenters. So, there's no guarantee that Edge server in on DC can connect to Origin in another DC using private IP address. It is recommended to use public IP addresses for CDN servers in this case.

2. Is there a problem if I use hold_ip_settings = true to configure a CDN over the Internet?

There'a no parameter hold_ip_settings in WCS configuration.

 

[ett]

Thanks, Max.

It is recommended to use public IP addresses for CDN servers in this case.

So, however, the cdn_ip could not be public IP address even if I and you want.
The cdn_ip will always be the local IP.

There'a no parameter hold_ip_settings in WCS configuration.

I know the parameter is not documented. I don't know that it is a documentation bug or not.
But it exists in bin/webcallserver as HOLD_IP_PROP=$(cat $WCS_CONFIG | grep "hold_ip_settings" | cut -d"=" -f2).

 

[Max]

I know the parameter is not documented. I don't know that it is a documentation bug or not.
But it exists in bin/webcallserver as HOLD_IP_PROP=$(cat $WCS_CONFIG | grep "hold_ip_settings" | cut -d"=" -f2).

The server launch script webcallserver is compatible with older WCS versions. In actual WCS build, this parameter can not be used.
The public and local IPs are always detected as Amazon recommends:

   PUBLIC_IP=$(curl -s -f http://169.254.169.254/latest/meta-data/public-ipv4)
   PUBLIC_IP_STATUS=$?
   LOCAL_IP=$(curl -s -f http://169.254.169.254/latest/meta-data/local-ipv4)
   LOCAL_IP_STATUS=$?

 

[ett]

Howdy Max.
First, hold_ip_settings actually exists at ami-07e6992e6c96a12ff, and it certainly works. WCS-5.2.597

$ curl -s -f http://169.254.169.254/latest/meta-data/ami-id
ami-07e6992e6c96a12ff

$ grep "hold_ip_settings" /etc/init.d/webcallserver
HOLD_IP_PROP=$(cat $WCS_CONFIG | grep "hold_ip_settings" | cut -d"=" -f2)

$ grep -E '^(ip|cdn_ip|hold)' /usr/local/FlashphonerWebCallServer/conf/flashphoner.properties
ip                     =0.0.0.0
ip_local               =0.0.0.0
cdn_ip                 =0.0.0.0
hold_ip_settings = true

$ sudo service webcallserver restart

$ grep -E '^(ip|cdn_ip|hold)' /usr/local/FlashphonerWebCallServer/conf/flashphoner.properties
ip                     =0.0.0.0
ip_local               =0.0.0.0
cdn_ip                 =0.0.0.0
hold_ip_settings = true

Second, my question was whether ORIGIN's cdn_ip could be 0.0.0.0 and EDGE in an external network could join like as

{ORIGIN on AWS us-east-1} --- through internet --- {EDGE on GCP asia-east2}
            |
            | through internet
            |
{EDGE on AWS eu-west-1}

I'm reading this article.

Dynamic CDN for Low Latency WebRTC Streaming | Flashphoner Streaming & Calls for Web

flashphoner.com

 

[ett]

When I implement that example on AWS and start WCS, the configuration file is rewritten by function check_amazon() as follows.

# Origin EU ( = o-eu1.flashponer.com :: The cdn_point_of_entry on the other five servers references this.)
cdn_ip=192.168.0.1
cdn_role=origin

# Edge 1 EU
cdn_ip=192.168.0.2
cdn_point_of_entry=o-eu1.flashponer.com
cdn_role=edge

# Edge 2 EU
cdn_ip=192.168.0.3
cdn_point_of_entry=o-eu1.flashponer.com
cdn_role=edge



# Origin US
cdn_ip=172.16.0.1
cdn_point_of_entry=o-eu1.flashponer.com
cdn_role=origin

# Edge 1 US
cdn_ip=172.16.0.2
cdn_point_of_entry=o-eu1.flashponer.com
cdn_role=edge

# Edge 2 US
cdn_ip=172.16.0.3
cdn_point_of_entry=o-eu1.flashponer.com
cdn_role=edge

### 'cdn_enabled=true' and 'cdn_nodes_resolve_ip=false' are omitted but they remain.

[o-eu1.flashponer.com] $ netstat -lnt
tcp        0      0 192.168.0.1:8084          0.0.0.0:*                   LISTEN    
tcp        0      0 192.168.0.1:3478          0.0.0.0:*                   LISTEN

Does WCS CDN work properly with this setting?

 

[Max]

Good day.

First, hold_ip_settings actually exists at ami-07e6992e6c96a12ff, and it certainly works. WCS-5.2.597

This parameter is used in launch script only to bypass IP detection on AMI first start.

I'm reading this article.

Dynamic CDN for Low Latency WebRTC Streaming | Flashphoner Streaming & Calls for Web

flashphoner.com

This article shows how to deploy a simple CDN on servers with public IPs, in Digital Ocean for example

Does WCS CDN work properly with this setting?

For this scheme to work, CDN nodes domain names should resolve to public nodes IPs, and port 8084 and media ports should be forwared to local interface on each node.

 

[ett]

Thank you for reply. I think I understand it somehow.
The CDN port 8084,3478 should be listened local ip of itself, isn't it?
And the other 5 servers(another 1 origin, 4 edges) will establish connection to the only 1 origin(= Origin EU = o-eu1.flashponer.com)'s 8443(at least not 8084,3478), right?
If the answer is yes, my question will be answered.


I think these cdn_ip=origin.flashponer.com,cdn_ip=edge.flashponer.com in the documentation are not good,
since they are rewritten to the local IP.
It looks external hostname rather than the internal hostname.

 

[Max]

And the other 5 servers(another 1 origin, 4 edges) will establish connection to the only 1 origin(= Origin EU = o-eu1.flashponer.com)'s 8443(at least not 8084,3478), right?

8084/tcp is used by default for CDN signaling. The connection is established "everyone to everyone", so this port should be available for all the servers on all the servers
8080/tcp is used for internal websocket connections to pull streams from Origin to Edge via WebRTC, so this port also should be available for all the servers on all the servers
WebRTC agent ports (34001-35000 by default) are used for CDN internal WebRTC media traffic, so this port range also should be available for all the servers on all the servers
8443/tcp is used by clients (browsers) to establish websocket connection for stream publishing or playing, so this port should be avalable on Origin for publishers and on Edge for subscribers
3478/tcp is the embedded TURN server port, so it sould be available for publishers and subscribers using TURN
Also, WebRTC media ports (media_port_from-media_port_to should be available on Origin for publishers and on Edge for subscribers
The cdn_point_of_entry parameter just sets for every server the address to connect to enter the CDN, i.e. the address for the first CDN signaling connection

 

[ett]

8084/tcp is used by default for CDN signaling. The connection is established "everyone to everyone", so this port should be available for all the servers on all the servers

Is it correct that each server only listens to its local ip?
- origin listen 192.168.0.1:8084 it can be established only from itself 192.168.0.1
- edge listen 192.168.0.2:8084 it can be established only from itself 192.168.0.2
[(origin)192.168.0.1]# nc -z 192.168.0.1 8084 => Connection to 192.168.0.1 8084 port [tcp/*] succeeded!
[(origin)192.168.0.1]# nc -z 192.168.0.2 8084 => FAIL
[(edge)192.168.0.2]# nc -z 192.168.0.1 8084 => FAIL
[(edge)192.168.0.2]# nc -z 192.168.0.2 8084 => Connection to 192.168.0.2 8084 port [tcp/*] succeeded!

As usual, EDGE gives an error.

12:47:43,570 INFO                     C - CDNOutbound-BOSS-pool-34-thread-1 Exception on channel [id: 0x4c65ddde], No route to host
12:47:43,571 ERROR             CDNNodes - Thread-55 Failed to connect to node {{origin_external_ip_OR_hostname}}
...
14:35:07,433 INFO                     C - CDNOutbound-BOSS-pool-34-thread-1 Exception on channel [id: 0x4b3919dd], No route to host
14:35:07,433 ERROR             CDNNodes - Thread-2017 Failed to connect to node {{origin_external_ip_OR_hostname}}

Failed to connect to node What is the port? 8084?
{{origin_external_ip_OR_hostname}} has listen 0.0.0.0:8080 and 192.168.0.1:8084

[(origin)192.168.0.1]# netstat -lnt | grep :808
tcp        0      0 0.0.0.0:8080                0.0.0.0:*                   LISTEN     
tcp        0      0 0.0.0.0:8081                0.0.0.0:*                   LISTEN     
tcp        0      0 0.0.0.0:8082                0.0.0.0:*                   LISTEN     
tcp        0      0 192.168.0.1:8084           0.0.0.0:*                   LISTEN     

[(edge)192.168.0.2]# nc -vz 192.168.0.1 8080-8084
Connection to 192.168.0.1 8080 port [tcp/webcache] succeeded!
Connection to 192.168.0.1 8081 port [tcp/tproxy] succeeded!
Connection to 192.168.0.1 8082 port [tcp/us-cli] succeeded!
nc: connect to 192.168.0.1 port 8083 (tcp) failed: No route to host
nc: connect to 192.168.0.1 port 8084 (tcp) failed: No route to host

[(edge)192.168.0.2]# nc -vz {{origin_external_ip_OR_hostname}} 8080-8084
Connection to {{origin_external_ip_OR_hostname}} 8080 port [tcp/webcache] succeeded!
Connection to {{origin_external_ip_OR_hostname}} 8081 port [tcp/tproxy] succeeded!
Connection to {{origin_external_ip_OR_hostname}} 8082 port [tcp/us-cli] succeeded!
nc: connect to {{origin_external_ip_OR_hostname}} port 8083 (tcp) failed: No route to host
nc: connect to {{origin_external_ip_OR_hostname}} port 8084 (tcp) failed: No route to host

 

[Max]

Please verify that there is a rule in iptables to accept connections to port 8084. (Or, try removing the REJECT (reject-with icmp-host-pro) rule.)

 

[ett]

Yes! I just noticed it and was about to report it!