HLS edge server not playing

[manikandan]

Hi, I am recieving error messages like this

12:51:48,000 ERROR             WCSAgent - API-ASYNC-pool-12-thread-1 7467ef3a-fe12-45b5-a875-822cd2ae370e Stream 38-1594642976-586182 local failed
12:51:48,039 ERROR               Server - HLS-38-1594642976-586182 Uncaught exception
java.lang.NullPointerException
    at com.flashphoner.media.output.hls.JavaHLSOutputWriter.run(Unknown Source)
12:51:50,154 ERROR             WCSAgent - API-ASYNC-pool-12-thread-1 9989a6fb-5226-4153-a8da-e75bc50cf4a1 Stream 38770-1594642332-460155 local failed
12:51:50,155 ERROR               Server - HLS-38770-1594642332-460155 Uncaught exception
java.lang.NullPointerException
    at com.flashphoner.media.output.hls.JavaHLSOutputWriter.run(Unknown Source)
12:51:53,296 ERROR             WCSAgent - API-ASYNC-pool-12-thread-1 e3196899-ae60-43d5-802c-fd6aaf51dd64 Stream 38-1594642976-586182 local failed
12:51:53,300 ERROR               Server - HLS-38-1594642976-586182 Uncaught exception
java.lang.NullPointerException
    at com.flashphoner.media.output.hls.JavaHLSOutputWriter.run(Unknown Source)
12:51:56,266 ERROR             WCSAgent - API-ASYNC-pool-12-thread-2 b9c91cae-5cad-4b51-8da1-fd16044a4a76 Stream 38770-1594642332-460155 local failed
12:51:56,296 ERROR               Server - HLS-38770-1594642332-460155 Uncaught exception
java.lang.NullPointerException
    at com.flashphoner.media.output.hls.JavaHLSOutputWriter.run(Unknown Source)
12:51:59,261 ERROR             WCSAgent - API-ASYNC-pool-12-thread-1 115d36c4-45ec-4609-9262-b34728d05ea9 Stream 38-1594642976-586182 local failed
12:51:59,265 ERROR               Server - HLS-38-1594642976-586182 Uncaught exception
java.lang.NullPointerException
    at com.flashphoner.media.output.hls.JavaHLSOutputWriter.run(Unknown Source)
12:52:02,266 ERROR             WCSAgent - API-ASYNC-pool-12-thread-2 7ff5d86a-028d-4e20-b60e-bdf8a14ff214 Stream 38770-1594642332-460155 local failed
12:52:02,267 ERROR               Server - HLS-38770-1594642332-460155 Uncaught exception
java.lang.NullPointerException
    at com.flashphoner.media.output.hls.JavaHLSOutputWriter.run(Unknown Source)

 

[manikandan]

That issue got resolved by restarting the webcall server. But I am continuouslu getting below error

13:07:21,509 ERROR             WCSAgent - API-ASYNC-pool-12-thread-3 ed6bef02-4c76-4680-8208-be4500c4324d Stream 32565-1594643557-764140 local failed
13:08:25,355 ERROR           MuxDecoder - STUN-UDP-pool-47-thread-3 Invalid RTCP packet received: total length should be multiple of 4 but is 82
13:08:33,983 ERROR             WCSAgent - API-ASYNC-pool-12-thread-3 db050e87-55d0-48b9-832a-b3ddd6f7c176 Stream 29131-1594645664-551483 local failed
13:13:43,724 ERROR           MuxDecoder - STUN-UDP-pool-47-thread-1 Invalid RTCP packet received: total length should be multiple of 4 but is 82

Also, when HLS streaming is played through Edge servers behind load balancers it returns CORS issue. I also enabled stickineess and it worked for few days but now suddently it shows CORS error now

 

[Max]

Good day.

But I am continuouslu getting below error

Please update to latest WCS build. Please note that CDN version is changed to 2.4 since build 5.2.685, so it is recommended to update all the CDN servers.

Also, when HLS streaming is played through Edge servers behind load balancers it returns CORS issue. I also enabled stickineess and it worked for few days but now suddently it shows CORS error now

You should add the following parameters to flashphoner.properties on HLS Edge servers behind load balancer

hls_access_control_headers=Access-Control-Allow-Origin: http://load-balancer-domain:8081;Access-Control-Allow-Methods: GET, HEAD;Access-Control-Max-Age: 3000;Access-Control-Expose-Headers: Accept-Ranges, Content-Range, Content-Encoding, Content-Length;Access-Control-Allow-Credentials: true

Please note that ACAO URL should be exactly the same as origin URL including port number.

 

[manikandan]

You should add the following parameters to flashphoner.properties on HLS Edge servers behind load balancer

I have already added like below in all edge servers

hls_access_control_headers=Access-Control-Allow-Origin: *;Access-Control-Allow-Methods: GET, HEAD;Access-Control-Max-Age: 3000;Access-Control-Expose-Headers: Accept-Ranges, Content-Range, Content-Encoding, Content-Length

Still CORS issue is happening. But when only one server is behind a load balancer it works without any issues. When addtional servers are added, it returns CORS error.

Even when I added the domain, I think flashphoner can't able to match the url because it returns the console erorr as below

Access to XMLHttpRequest at 'https://LB-domainname:port/teststream/teststream.m3u8?aclAuth=test' from origin 'http://localhost' has been blocked by CORS policy: The 'Access-Control-Allow-Origin' header contains the invalid value 'http'.

 

[manikandan]

I fixed the issue by updating to latest build but after update, WCS CDN failed to push streams from origin to edge servers. Any additional configurations is need to be done with the latest build?

When I logged in as admin and ran "show cdn-nodes", it doesn't show the edge servers, same as in edge servers. So CDN seems to be not working. But it worked perfectly in my previous build

FYI, My previous build version is 5.2.634 and my implementation is based on CDN2.0

Also after update, I can't see any cdn commands mentioned here https://docs.flashphoner.com/display/WCS52EN/CDN+2.4

 

[Max]

CDN commands are part of CLI v2.
CLI v2 is located on port 2002 by default.

Any additional configurations is need to be done with the latest build?

No additional configuration required.

Please share your Origin - Edge configurations (flashphoner.properties) or provide SSH access to these two servers. We will check.

 

[manikandan]

Thanks Max. Please check the config files below
Origin server

#server ip
ip                     ={PUBLIC_IP}
ip_local               ={LOCAL_IP}

#webrtc ports range
media_port_from        =31001
media_port_to          =32000

#codecs
codecs                   =opus,alaw,ulaw,g729,speex16,g722,mpeg4-generic,telephone-event,h264,vp8,flv,mpv
codecs_exclude_sip       =mpeg4-generic,flv,mpv
codecs_exclude_streaming =flv,telephone-event
codecs_exclude_sip_rtmp  =opus,g729,g722,mpeg4-generic,vp8,mpv

#websocket ports
ws.port                 =8080
wss.port                =8443

client_mode=false

rtc_ice_add_local_component=true
rtc_ice_add_local_interface=true


#rest-api

disable_rest_auth=true
rest_access_control_allow_origin=*
rest_access_control_allow_headers=content-type,x-requested-with
rest_access_control_allow_methods=POST

#CDN
cdn_enabled=true
cdn_ip={LOCAL_IP}
cdn_role=origin
cdn_port=8084
cdn_nodes_timeout=60

# disable web interface

http_enable_paths=rest,action,shared,embed_player,empty

Edge server:

#server ip
ip                     ={PUBLIC_IP}
ip_local               ={LOCAL_IP}

#webrtc ports range
media_port_from        =31001
media_port_to          =32000

#codecs
codecs                   =opus,alaw,ulaw,g729,speex16,g722,mpeg4-generic,telephone-event,h264,vp8,flv,mpv
codecs_exclude_sip       =mpeg4-generic,flv,mpv
codecs_exclude_streaming =flv,telephone-event
codecs_exclude_sip_rtmp  =opus,g729,g722,mpeg4-generic,vp8,mpv

#websocket ports
ws.port                 =8080
wss.port                =8443

client_mode=false

rtc_ice_add_local_component=true
rtc_ice_add_local_interface=true

#HLS
client_acl_property_name=aclAuth
hls_access_control_headers=Access-Control-Allow-Origin: http://localhost;Access-Control-Allow-Methods: GET, HEAD;Access-Control-Max-Age: 3000;Access-Control-Expose-Headers: Accept-Ranges, Content-Range, Content-Encoding, Content-Length;Access-Control-Allow-Credentials: true

#rest-api

disable_rest_auth=true
rest_access_control_allow_origin=*
rest_access_control_allow_headers=content-type,x-requested-with
rest_access_control_allow_methods=POST

#CDN

cdn_enabled=true
cdn_ip={LOCAL_IP}
cdn_point_of_entry={ORIGIN_PUBLIC_IP}
cdn_role=edge
cdn_port=8084
cdn_nodes_timeout=60

# disable web interface

http_enable_paths=rest,action,shared,embed_player,empty

 

[Max]

Please check iptables output

sudo iptables -L -n

If there are some rules and no port 8084 among them, just flush all the rules

sudo iptables -F

It is not necessary to block ports on instance level because security group rules are working on hosting level.
If this does not help please provide SSH access to these two servers. We will check.

Even when I added the domain, I think flashphoner can't able to match the url because it returns the console erorr as below

You should add domain name and port because browser (headers are analyzed on browser level, not on server) strictly compares ACAO header with request header.
So yuo should set ...Access-Control-Allow-Origin: http://localhost:8081;... for example.

 

[manikandan]

You should add domain name and port because browser (headers are analyzed on browser level, not on server) strictly compares ACAO header with request header.

Yeah. I fixed that. I hope that is not the issue with CDN connections.

If this does not help please provide SSH access to these two servers. We will check.

Sorry Max. I can't give SSH access due to privacy concerns. I will check with the steps you shared and come back to you again.

 

[manikandan]

You should add the following parameters to flashphoner.properties on HLS Edge servers behind load balancer

hls_access_control_headers=Access-Control-Allow-Origin: http://load-balancer-domain:8081;Access-Control-Allow-Methods: GET, HEAD;Access-Control-Max-Age: 3000;Access-Control-Expose-Headers: Accept-Ranges, Content-Range, Content-Encoding, Content-Length;Access-Control-Allow-Credentials: true

Please note that ACAO URL should be exactly the same as origin URL including port number.
[/QUOTE]

@Max Is it possible to add wildcards in the Access-Control-Allow-Origin, like https://*.example.com

 

[Max]

Is it possible to add wildcards in the Access-Control-Allow-Origin, like https://*.example.com

Unfortunately, no. This is Chrome oversecurity issue. The domain name, protocol and port should be exactly the same as request source.

 

[manikandan]

@Max I have another question. We have an android app where HLS stream can be played by the users. If the ACAO has to be set with domain name to whitelist, it will not work with mobile app. right?
Is there any way we can do the same for mobile apps?

 

[Max]

I have another question. We have an android app where HLS stream can be played by the users. If the ACAO has to be set with domain name to whitelist, it will not work with mobile app. right?
Is there any way we can do the same for mobile apps?

Usually, mobile applications uses WebView to play HLS. So it depends on WebView implementation on a certain platform, but it behaviour should not differ from default browser.
If, however, you 've created mobile application from scratch including HTTP custom implementation, you have to implement ACAO headers support too.

 

[manikandan]

okay. I am checking on that.
Also is it possible to add multiple domains to whitelist in the flashphoner properties file.
If it is not, can you add that feature because streaming will be played from various web apps, so adding new server for each web app will not be a good approach. Requesting to implement that feature.

 

[Max]

Also is it possible to add multiple domains to whitelist in the flashphoner properties file.
If it is not, can you add that feature because streaming will be played from various web apps, so adding new server for each web app will not be a good approach. Requesting to implement that feature.

At StackOverflow supposed the whitespace to be divider in ACAO header, please try it

hls_access_control_headers=Access-Control-Allow-Origin: http://load-balancer-domain:8081 http://load-balancer2-domain:8081 http://load-balancer3-domain:8081;...

If this does not help we recommend to do the following:
1. Set up nginx as reverse proxy for HLS as described here
2. Set up nginx to return ACAO headers depending on origin domain as described here or here

 

[Max]

If nothing helps or reverse proxy solution looks too complex, we raised the ticket WCS-2807 to implement a setting that will enable to return HTTP origin request URL in ACAO header. We'll let you know in this topic about results.

 

[manikandan]

okay. Thank you Max.

I have one more question, when a stream is published in origin server and what is the maximum time needed for edge server to retrieve that stream from origin?
Will the edge server pull the stream immediately from origin server?

 

[Max]

I have one more question, when a stream is published in origin server and what is the maximum time needed for edge server to retrieve that stream from origin?
Will the edge server pull the stream immediately from origin server?

Edge server pulls a stream from Origin server only if someone tries to play the stream on Edge. This reduces inter-server traffic, it may be meaningful when Origin and Edge servers are in different datacenters on different continents.
The maxumum time to retrieve the stream on Edge depends on stream resolution, bitrate and inter-server channel quality. Also, a subscriber will see the stream after first key frame receiving, so stream keyframe interval also affects actual speed of stream viewing.

 

[manikandan]

But the edge server will show the active CDN routes once the stream is published in origin server.
I mean when this REST call "rest-api/cdn/show_routes" is executed and the stream is published in origin server, even if no one tries to play the stream on edge that API call will show the stream routes. right?

Because I am doing ACL on stream, to do that I need the Origin server IP where the stream got published. I am using above REST API call to fetch that serverIP. Sometimes the server IP not get returned after the stream is published in origin server.
Is that there any other way or any other API to get the published stream server IP?

 

[Max]

But the edge server will show the active CDN routes once the stream is published in origin server.
I mean when this REST call "rest-api/cdn/show_routes" is executed and the stream is published in origin server, even if no one tries to play the stream on edge that API call will show the stream routes. right?

Stream route is a possible path for edge to pull the stream from origin. It can be build right after stream is published on Origin.

Because I am doing ACL on stream, to do that I need the Origin server IP where the stream got published. I am using above REST API call to fetch that serverIP. Sometimes the server IP not get returned after the stream is published in origin server.
Is that there any other way or any other API to get the published stream server IP?

Why you do not know server IP where stream is published? Do you have autoscaling group of Origins behind a load balancer?
You can use REST hook /publishStream, in this case you will always know Origin IP and stream published name on backend server.