Scaling large recording app

[Evert Van Cauwenberg]

Hi there

We are currently a large scale recording application and we are in the final delivery.
Currently we are using this rest api methods to take a snapshot, start and stop:
* /stream/snapshot
* /stream/startRecording
* /stream/stopRecording

We have currently deployed flashphoner on a t2.medium on AWS but what are the recommendations for a production environment (load balanced via ELB) where we expect to receive 5000-10000 recordings?

We also are struggling with a very low quality snapshot (see screenshot), i have already searched the forum and increased the bitrate and heap.
Our flashphoner.properties:

#server ip
ip                       =
ip_local               =

#webrtc ports range
media_port_from        =31001
media_port_to          =32000

# codecs
codecs                         =opus,alaw,ulaw,g729,speex16,g722,mpeg4-generic,telephone-event,vp8,h264,flv,mpv
codecs_exclude_sip       =mpeg4-generic,flv,mpv
codecs_exclude_streaming =flv,telephone-event
codecs_exclude_sip_rtmp  =opus,g729,g722,mpeg4-generic,vp8,mpv

#websocket ports
ws.port                 =8080
wss.port                =8443

# Custom
on_record_hook_script    =on_record_hook.sh
client_mode=false
rtc_ice_add_local_component=true

streaming_video_decoder_fast_start=false
webrtc_cc2_twcc=false
webrtc_cc_min_bitrate=1000000
webrtc_cc_max_bitrate=2000000

#ice_tcp_transport=true
#ice_tcp_send_buffer_size=1048576
#ice_tcp_receive_buffer_size=1048576

Kind regards
Evert

 

[Max]

Hello
Snapshot is built on the stream from web cam.
So it has the same resolution as stream at the moment of taking snapshot.
Please check out charts from chrome://webrtc-internals
What resolution do you see there?
As you can see, stream resolution in my case is 320x240. So snapshot quality will be 320x240 too and will be bad if stretched.

If you test 1280x720 you should see a good bitrate about 2Mbps and good 1280x720 picture.
If you have not enough network channel your resolution and bitrate can be decreased down to low values.

 

[Max]

Regarding production requirements.
Minimal production requirements:

1 CPU Xeon, 8-10 cores, 32 GB RAM

High load production

2 CPU Xeon, 8-10 cores, 64 GB RAM and more

So you would need a pool of production servers to get this working.
If you use AWS you would match hardware requirements above to an AWS instance.

where we expect to receive 5000-10000 recordings?

This is too high volume. You may encounter AWS iOPS limits. Try to consider instances with fast NVM volumes.
You can also setup a load test using re-publishing API. You can re-publish N streams from one server to another server and record these streams by REST API.
https://docs.flashphoner.com/display/WCS52EN/To+another+WCS+server+via+WebRTC

 

[Evert Van Cauwenberg]

Tnx for the update Max

Is it possible to use ELB to load balance between 1-2...4 instances? Since i store all the recordings on S3?

 

[Max]

We didn't test with AWS ELB.
However ELB should work because it uses common HTTP/HTTPS connections and WCS server uses ws/wss connections (Websocket)
To setup health checking you can use statistic URL http://host:8081?action=stat
https://docs.flashphoner.com/display/WCS52EN/Load+and+resource+usage+information
If you have any troubles with ELB setup, please let us know. We will re-check it.

 

[Evert Van Cauwenberg]

Hi Max

Thnx for your reply, i'll test it today!

For the snapshot please check the screenshot of the canvas, the snapshot and the recorded stream.
https://s3-eu-west-1.amazonaws.com/newfusion-temp/Screen+Shot+2019-03-28+at+10.05.09.png (canvas screenshot)
https://s3-eu-west-1.amazonaws.com/newfusion-temp/Screen+Shot+2019-03-28+at+10.05.30.png (snapshot)
https://s3-eu-west-1.amazonaws.com/newfusion-temp/Screen+Shot+2019-03-28+at+10.07.55.png (Webrtc screenshot)
https://s3-eu-west-1.amazonaws.com/newfusion-temp/Untitled.m4v recording

Thnx in advance!

Kind regards!
Evert