Can you please link ticket WCS-2753 to my account

Max

Administrator
Staff member
Good day.
We'll let you know in this topic on this ticket updates.
 

Azhar Ali

Member
Hi Max,

Thanks for that, Any idea on time estimate on that so we can plan it, anything you can do to make it faster would be highly appreciated..

thanks
Azhar
 

Max

Administrator
Staff member
Good day.
We've set up sample CDN of 1 Origin and autoscaling group of Edges. Now we working on load balancing setup. Thank for your patience.
 
Hello!

We are excited with this tutorial and try it!

Bellow our feedback:
(NOTE: our origins servers are not located @google)

4. Create HTTP legacy health check. Note that load balancer always checks server root “/”
With this setting we got unhealth instance.
We are forced to change from "/" to "/health-check" and then we got a green flag as health instance on load balance.

5. Create frontend configuration for HTTP(S) WCS ports and WS(S) WSC ports. Select “Create static IP” and create static IP address for load balancer
Missing port 8445 for HLS stream

We got the edge over load balance up and running like a charm.
We just add our tunning settings as recommended on documentation before take an edge image for loadbalance.

Questions:

a-) Why using Java 12 and not 1.8 as default?
b-) How can I simulate load to reach 80% of CPU to check if new servers are going up and running?

Thanks!
 

Max

Administrator
Staff member
Good day.
4. Create HTTP legacy health check. Note that load balancer always checks server root “/”
With this setting we got unhealth instance.
We are forced to change from "/" to "/health-check" and then we got a green flag as health instance on load balance.
Please note you should add the following parameter to flashphoner.properties file of Edge instance
Code:
http_enable_root_redirect=false
for "http://wcs:8081/" health checking to work.
By default, HTTP GET request to WCS root returns 302 Found to redirect to web admin interface.
5. Create frontend configuration for HTTP(S) WCS ports and WS(S) WSC ports. Select “Create static IP” and create static IP address for load balancer
Missing port 8445 for HLS stream
The manual is common and describes WebRTC Edges only. You can add any necessary ports to the load balancer.
a-) Why using Java 12 and not 1.8 as default?
Java 8 is good enough for development but may be not so stable under high load. That's why we recommend to use JDK 12 or 14 and ZGC for high loaded servers.
b-) How can I simulate load to reach 80% of CPU to check if new servers are going up and running?
We've tested it by publishing WebRTC stream with certain resolution (HD or FullHD if channel bandwidth allows) and playing it with another one, this will enable transcoding on edge server and will load CPU over 80% depending on encoders quantity.
 

Azhar Ali

Member
Hi Max,

I followed the document and deployed the setup. I am not sure how to set up SSL on the edge as domain is linked with the external IP of the load balancer. I have normally used the cert-auto tool and certified the domain using letsencrypt, download the pem files, and upload them using the admin.

The second question is, I used the console application and generated the load, after 80% cpu a new instance was created but no traffic was going to it. I used to http://wcs-edge-1:8888/?action=stat and http://wcs-edge-2:8888/?action=stat to see the webrtc connections. I generated pull streams of 1000 but it was never going above 400

Main thing I want to test is, one stream is published and our users grow, it can scale up. Anything you can suggest?
 

Max

Administrator
Staff member
Good day.
I followed the document and deployed the setup. I am not sure how to set up SSL on the edge as domain is linked with the external IP of the load balancer. I have normally used the cert-auto tool and certified the domain using letsencrypt, download the pem files, and upload them using the admin.
Sorry, we missed this moment in doc. You should set up SSL right after creation of Origin and Edge instances:
1. Create server instance
2. Install, configure and start WCS
3. Get the SSL certificate and import it to WCS. It is recommender to use wildcard domain certificate.
Then you prepare Edge disk image which will contain certificates with other Edge settings
We do not recommend to use Lets'Encrypt in this case because it requires to renew certificates every 3 months, so you have to stop all the instance group, update Edge disk image and start instance group again.
The second question is, I used the console application and generated the load, after 80% cpu a new instance was created but no traffic was going to it.
This is the load balancer issue. It uses Client IP and protocol to bind requests from one client to one server. So you have to test from two different IPs to make sure requests go to a new instance.
I used to http://wcs-edge-1:8888/?action=stat and http://wcs-edge-2:8888/?action=stat to see the webrtc connections. I generated pull streams of 1000 but it was never going above 400
The default media ports range (31001-32000) allows no more than 400 WebRTC connections. So you have to expand this range, update project firewall rules and Edge disk image.
To update Edge disk image, you should:
1. Disable autoscaling and delete all instances from instance group
2. Delete Edge disk image
2. Start Edge instance from which this image was created (let's name it master Edge instance)
3. Expand media ports range (or update some other Edge settings)
4. Stop Edge instance
5. Create new Edge disk image
6. Enable autoscaling of instance group
The procedure is the same for updating Edge settings or SSL certificates
 

Max

Administrator
Staff member
Would you not just install the SSL on the load balancer as the domain will be linked with LB IP
Unfortunately, no. There is a three main types of Load balancers in Google Cloud Platform:
1. HTTP(S) Load balancer. It supports SSL management, but destination ports can be only 80, 443 and 8080. Also, it cannot forward requests to different instance ports based on protocol prefix (http, https, ws, wss), it can only forward by domain name or folder. So this is enough for web services, but not for broadcasting
2. TCP/SSL Proxy. The same drawback, this solution is intended to mail and database services.
3. TCP/UDP Network Load balancer. This type supports any port forwarding, but does not support SSL certificate management, it just passes all the traffic through without any de/encription. We have to choose this type because of port forwarding flexibility.
So, we should install SSL certificate on instance end.
 
1 - create a node edge with SSL like edge.yourdomain.com
2 - install it on WCS as described
3 - point LOAD BALANCE IP to like edge.yourdomain.com
4 - no matter how many edges do you start, all of them has the same domain name and SSL and also the same image from WCS settings.
 

Max

Administrator
Staff member
1 - create a node edge with SSL like edge.yourdomain.com
2 - install it on WCS as described
3 - point LOAD BALANCE IP to like edge.yourdomain.com
4 - no matter how many edges do you start, all of them has the same domain name and SSL and also the same image from WCS settings.
You're right.
We prepared the online doc about GCP load balancing with autoscale to clarify the isues mentioned in this topic.
 

Azhar Ali

Member
Thanks, Setup is all done and working nicely. We have tested this with live environment by changing the auto-scale CPU to 15% which caused new instances to start and started taking the new connections.
 
Today we experience an issue with Google Cloud.

An instance has been "shutdown" from LB but users were connected to them.

As work arround we edited the Instance Group and change to INCREASE ONLY.
Decrease will be manually done until we find another way.
 

Max

Administrator
Staff member
An instance has been "shutdown" from LB but users were connected to them.
This is known autoscaling issue, it can be reproduced in AWS LB with autoscaling too.
We cannot prevent instance shutdown from inside the instance. So manual instance stopping seems to be the only solution for any autoscaling configuration.
 
We are attempting to use Google API to monitor that instance and check if there is users connect to .. if not, shutdown from LB.

Thanks Max!
 
Top