CDN - failed to connect to origin server

[manikandan]

Hi,
I am trying to do live stream using CDN. I have two load balancers (AWS), one for origin and another for edge. I followed the steps with dynamic CDN and I am getting
"Failed to connect error" in edge server.

Edge:
cdn_enabled=true
cdn_ip={ip_local}
cdn_point_of_entry=origin.demo.com
cdn_role=edge

Origin:
cdn_enabled=true
cdn_ip={ip_local}
cdn_role=origin

1) So how this edge server connects with origin server and which ports needs to be opened for edge to connect to Origin?
2) And is there any limit on number of origins and edge servers behind the load balancer?

 

[Max]

1) So how this edge server connects with origin server and which ports needs to be opened for edge to connect to Origin?

CDN internal signaling is going on TCP port 8084.
So this port must be open on each CDN node.

cdn_port = 8084

Media traffic (audio + video) works via UDP ports

media_port_from = 31001
media_port_to = 32000

So UDP range [31001-32000] should be open too.

https://docs.flashphoner.com/display/WCS52EN/Settings+file+flashphoner.properties

2) And is there any limit on number of origins and edge servers behind the load balancer?

No such limits.

 

[manikandan]

Thanks Max for the quick response. Just to confirm, is the above Edge and Origin server configuration is correct?
I am getting following error when I checked with the server logs

java.lang.IllegalArgumentException: empty text
at org.jboss.netty.handler.codec.http.HttpVersion.<init>(Unknown Source)
at org.jboss.netty.handler.codec.http.HttpVersion.valueOf(Unknown Source)
at org.jboss.netty.handler.codec.http.HttpRequestDecoder.createMessage(Unknown Source)
at org.jboss.netty.handler.codec.http.HttpMessageDecoder.decode(Unknown Source)
at org.jboss.netty.handler.codec.http.HttpMessageDecoder.decode(Unknown Source)
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(Unknown Source)
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(Unknown Source)
at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
at org.jboss.netty.channel.socket.nio.NioWorker.read(Unknown Source)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(Unknown Source)
at org.jboss.netty.channel.socket.nio.DeadlockAwareNioWorker.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
16:31:45,038 ERROR HttpServerHandler - HTTP-pool-2-thread-1 HTTP error
java.lang.IllegalArgumentException: empty text
at org.jboss.netty.handler.codec.http.HttpVersion.<init>(Unknown Source)
at org.jboss.netty.handler.codec.http.HttpVersion.valueOf(Unknown Source)
at org.jboss.netty.handler.codec.http.HttpRequestDecoder.createMessage(Unknown Source)
at org.jboss.netty.handler.codec.http.HttpMessageDecoder.decode(Unknown Source)
at org.jboss.netty.handler.codec.http.HttpMessageDecoder.decode(Unknown Source)
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(Unknown Source)
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.cleanup(Unknown Source)
at org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(Unknown Source)
at org.jboss.netty.channel.Channels.fireChannelDisconnected(Unknown Source)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(Unknown Source)
at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(Unknown Source)
at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(Unknown Source)
at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(Unknown Source)
at org.jboss.netty.channel.Channels.close(Unknown Source)
at org.jboss.netty.channel.AbstractChannel.close(Unknown Source)
at com.flashphoner.server.http.F.exceptionCaught(Unknown Source)
at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(Unknown Source)
at org.jboss.netty.channel.Channels.fireExceptionCaught(Unknown Source)
at org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(Unknown Source)
at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
at org.jboss.netty.channel.socket.nio.NioWorker.read(Unknown Source)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(Unknown Source)
at org.jboss.netty.channel.socket.nio.DeadlockAwareNioWorker.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

And to get HLS URL, we can use the Edge domain url with port 8082. right?

 

[Max]

Default HLS ports

hls.http.port = 8082
hls.https.port = 8445

Vaild HLS URLs

http://edge.mycompany.com:8082/stream1/stream1.m3u8

https://edge.mycompany.com:8445/stream1/stream1.m3u8

Here stream1 is stream name

Do you play correct URLs? Please attach full logfile.

 

[manikandan]

Yes. I am playing the correct urls with port (8082). But I am getting this error.
Please see the log file

 

[Max]

The log fragment attached seems like you're trying to play HTTPS via HTTP port or vice versa.
Aslo, please check if the same stream is playing correctly from Edge using WebRTC player. If not, please collect reports from both Origin and Edge as described here and send to support@flashphoner.com. SSH access to CDN would also be useful.

 

[manikandan]

I figured out the issue. I have enabled acl_auth and it seems when Edge URL tries to authenticate from my backend server, it is breaking something.
Is there any guidelines to follow acl_auth for edge servers?

But still I am seeing errors in the server_logs/flashphoner.log file. This error in edge server comes after /playHLS call

 

[manikandan]

I am seeing some other errors like

06:56:28,837 ERROR HttpServerHandler - HTTP-pool-2-thread-97 HTTP error
java.lang.IllegalArgumentException: invalid version format: DAￌ￶￈>:ﾝ=5ﾄﾜ</A
at org.jboss.netty.handler.codec.http.HttpVersion.<init>(Unknown Source)
at org.jboss.netty.handler.codec.http.HttpVersion.valueOf(Unknown Source)
at org.jboss.netty.handler.codec.http.HttpRequestDecoder.createMessage(Unknown Source)
at org.jboss.netty.handler.codec.http.HttpMessageDecoder.decode(Unknown Source)
at org.jboss.netty.handler.codec.http.HttpMessageDecoder.decode(Unknown Source)
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(Unknown Source)
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(Unknown Source)
at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
at org.jboss.netty.channel.socket.nio.NioWorker.read(Unknown Source)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(Unknown Source)
at org.jboss.netty.channel.socket.nio.DeadlockAwareNioWorker.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

 

[Max]

I figured out the issue. I have enabled acl_auth and it seems when Edge URL tries to authenticate from my backend server, it is breaking something.
Is there any guidelines to follow acl_auth for edge servers?

Please read the documentation about CDN ACL. It is impossible to use CDN ACL and HLS backend authentication simultaneously. So disable HLS authentication in CDN:

hls_auth_enabled=false

 

[manikandan]

@Max Similar to playHLS api aithentication from backend server, how can I authenticate CDN streaming (HLS) at viewer side using my backend authorization?

 

[Max]

how can I authenticate CDN streaming (HLS) at viewer side using my backend authorization?

To authenticate HLS (and, of course, any other streaming) in CDN, you should:
1. Apply ACL to stream on Origin server from backend server.
2. When user connects to your frontend, pass user access credentials to backend server.
3. Backend server sends to frontend ACL key to play HLS stream, key1 for example.
4. HLS player on frontend requests the stream bu URL https://edge:8445/streamName/streamName.m3u8?aclAuth=key1

 

[manikandan]

1. Apply ACL to stream on Origin server from backend server.

The viewers will get dynamic keys on the fly so applying keys on a stream when published is not possible.

And is there any expiration time configuration to view any stream?

 

[Max]

The viewers will get dynamic keys on the fly so applying keys on a stream when published is not possible.

1. Assume you issue 100 access keys: [K1 .. K100]
2. User1 opens the HLS player page and uses aclKey=K1
3. User2 opens the HLS player page and uses aclKey=K2
4. User3 opens the HLS player page and uses aclKey=K3
etc.

So your back-end aware about the key pool [K1 .. K100] and your back-end can control this pool any time using /add /remove operations.

The key is arbitrary string. So it can be user login or hash(userLogin).
If so you just apply all authenticated user names as keys in the beginning.

 

[Max]

And is there any expiration time configuration to view any stream?

There are no expiration time for ACl list. If you add a key, this key will allow access. You have to remove this key immediately or cleanup ACL list to restrict access.

Example of scenario:
1. User1 is trying to play stream1 with key "key1". Stream is failed to play.
2. Admin adds the "key1" to ACL list for "stream1".
3. User1 is trying to play stream1 with key "key1". Playback success.
4. Admin removes the "key1" from ACL list for "stream1".
5. User1 playback failed.
6. Admin adds again the "key1" to ACL list for "stream1".
7. User1 is trying to play stream1 with key "key1". Playback success.

 

[manikandan]

Great. Got some idea. Thanks Max.

4. HLS player on frontend requests the stream bu URL https://edge:8445/streamName/streamName.m3u8?aclAuth=key1

Is the CDN ACL url should be https(8445) or http(8082) can be used?
https://flashphoner.com/docs/wcs5/w...e_conf_directory-server_properties_config.htm

 

[Max]

You can use either https 8445 or http 8082.
However some browsers may restrict unsecure http requests from secure https pages.
So if you play http HLS from https page it can be an issue.

 

[manikandan]

Thank you. Let me check.
And is there any fix for above post https://forum.flashphoner.com/threads/cdn-failed-to-connect-to-origin-server.12874/#post-22775

It seems the issue comes with netty java package regarding HTTPS access. I am not accessing any HTTPS in the server.
This issue came only after enabling CDN (origin and edge servers). I am seeing lot of errors now.

 

[manikandan]

Max, I figured out the issue with error logs. Thanks for your support and quick response.

 

[manikandan]

Max, I have another question regarding CDN HLS.
As said, I have enabled CDN aclAuth and it is working fine. But using Rest-api, I am trying to get the number of active viewers for a HLS stream and I can't able to get it.
Is there an API to fetch the active HLS viewers in Edge servers?

 

[Max]

Good day.

Is there an API to fetch the active HLS viewers in Edge servers?

No, there's no API to detect how much viewers per HLS stream.
You can count HLS viewers on backend while autenticating them.