AWS Auto Scaling with Stream Recording

[Taylor]

Hello

We've been setting up the WCS servers to sit behind a Load Balancer and Auto Scaling Group and found an issue when dealing with scaling down. We don't think it's solvable based on the requirements that we want, however we decided to post it here in case anyone has a solution.

So with our WCS servers we want them to handle stream recording, where the users would stream their video from our website onto the WCS server via the Load Balancer DNS. The servers would also include a mounted shared file storage where the videos are stored so that any WCS server can be used to fetch any of the recorded videos.

We've been able to successfully setup everything on AWS, launch the WCS servers and do stream recording to them and play them back without issue.

However when it comes to scaling down WCS servers, there's an issue that we've been trying to resolve:

E.g. Say we have 5 WCS servers running, each with at least 1 user streaming to it at any given time. We now want to scale down to 3 servers but we don't want to disconnect any of the existing users' streams.
How do we scale-down without disconnecting the users' stream?

Since we don't want to immediately terminate the WCS server, we've set the Auto Scaling Group policy to be protected from scaling-in, stopping the automatic scaling down, and allows us to manually handle scaling down instead.

The way we thought to do it was to detach the WCS server from the Auto Scaling group, which allows the server to continue to run but no longer sit under the Load Balancer.

However when we went to test this it didn't exactly work:

• We logged onto the WCS server.
• We navigated to the Stream Recording page and started a stream using the Load Balancer DNS.
• Once it started, we confirmed which server was handling the recording by checking the /?action=stat page on the individual servers.
• Once we identified the server handling the stream, we went to AWS Auto Scaling Group and detached that instance.
• During the period it took to detach, approx. 5 minutes, we were able to continue streaming without issue.
• Once the detach process finished, the session almost immediately disconnects itself and stops the stream recording entirely.

Of note, we can still access the WCS server by it's public IP address directly and can do stream recording through that.

We're hoping that there's a solution here that you can suggest to us, but we have a feeling that we cannot use the Auto Scaling Group and Load Balancer for our purpose of stream recording, at least without encountering issues during scaling down.

Thank you in advance.

 

[Max]

Good day.
When you're connecting through Load Balancer DNS, the client uses its endpoint to establish a websocket session. Not a server endpoint directly. Then, you're detaching the server from the Load Balancer. When detaching is finished, Load Balancer closes the websocket connection because it will not work any more with the detached server. Therefore, all the clients using Load Balancer endpoint to connect to the server will be disconnected after the server is detached. This is the logic of AWS Load Balancer.
You can only scale down the server which has no publishing clients at the time. So you should periodically check the server load and detach it only when all the clients left it.