Streaming video quality issue with 5.2.639 version

[jyotipatidar]

Hi,
We are observing blur video on live streaming as well as stored video (s3)

We followed one of your thread and changed the order of VP8, as below

codecs=opus,alaw,ulaw,g729,speex16,g722,mpeg4-generic,telephone-event,vp8,h264,flv,mpv

This has improved the live video quality but the stored video lost the voice, video is getting saved without voice.
Please help here we need it urgently.

Thanks

 

[Max]

Good day.

We are observing blur video on live streaming as well as stored video (s3)

This is probably bitrate drop known issue. You should raise publishing bitrate on client side (preferrable) or on server side (less flexible) as described here. For example, set the minimal bitrate for 240p to 200 kbps:
- on client

publishStream = session.createStream({
    name: streamName,
    display: localVideo,
    constraints: {
           video: {
                 width: 320,
                 height: 240,
                 minBitrate: 200
                 maxBitrate: 500
           }
    }
).on(STREAM_STATUS.PUBLISHING, function (publishStream) {
    ...
});
publishStream.publish();

- on server

webrtc_cc_min_bitrate=200000
webrtc_cc_max_bitrate=500000

This has improved the live video quality but the stored video lost the voice, video is getting saved without voice.

To switch to VP8 correctly, you should also disable H264 codec on publishin client

publishStream = session.createStream({
    ...
    stripCodecs: "H264,h264"
}).on(STREAM_STATUS.PUBLISHING, function (publishStream) {
    ...
});
publishStream.publish();

We recommend to switch to VP8 only if publisher does not support H264 because this may lead to server transcoding for players which do not support VP8 (Safari for example)

 

[jyotipatidar]

Thank you so much max, this has improved the video quality, we are still in testing mode, will come back to you if any further issue on this.
Also would you recommend upgrading to 5.2.650 version from 6.2.639 ? Will it help in improving the things ?

 

[jyotipatidar]

Hi Max,
1) After doing the suggested change on client side
width: 320,
height: 240,
minBitrate: 200
maxBitrate: 500

Streaming quality is good but stored video (s3) is still blur.
We then tried to change server side as well
webrtc_cc_min_bitrate=200000
webrtc_cc_max_bitrate=500000

And still the s3 stored video quality remains blurry.

2) Also some times flashphoner stops saving the video on records folder (we need to restart the server in this case )

Please suggest

 

[jyotipatidar]

Here I'm consolidating all the issues on which we need your help, as we are newbie
1) Some times recording does not work (it automatically stops recording it shows 0 byte in /records folder). General observation is when recording is more then 30 min or more the video doesn't even save to records folder - We need immediate solution for this
2) After doing the suggested change (in above post by Max) on client side Streaming quality is good but stored video (s3) is still blur.
3) Voice lagging in streaming video (voice and video sync up is not up to the mark) and some time it drops the voice from recorded video
4) Need to enable full screen on the playing video (player) (Android Chrome browser)
5) Need to break video in parts instead of saving it as 1 big mp4 file in s3
6) We need compression while storing video on s3
7) Need the statistics like how many concurrent broadcast can run at a time with how many users (with EC2 c5.large instance)
8) Need the statistics like how many concurrent webinar can run at a time with how many users (with EC2 c5.large instance)

Thanks in advance, but would appreciate a quick response !

 

[Max]

Hello

We have checked your instance.

1) Some times recording does not work (it automatically stops recording it shows 0 byte in /records folder). General observation is when recording is more then 30 min or more the video doesn't even save to records folder

We checked recording logs. It looks like video traffic issue.

10:00:13,571 WARN  ileRecordAudioBuffer - CommonFileRecorderThread Full buffer and no video packets, dropping packets
10:00:13,571 WARN  ileRecordAudioBuffer - CommonFileRecorderThread Full buffer and no video packets, dropping packets
10:00:13,571 WARN  ileRecordAudioBuffer - CommonFileRecorderThread Full buffer and no video packets, dropping packets

Solution: try to switch to TCP protocol to avoid high packet lost.

Example:

session.createStream({
    name: streamName,
    display: localVideo,
    cacheLocalResources: true,
    receiveVideo: false,
    receiveAudio: false,
    transport: "TCP"
}).on(STREAM_STATUS.PUBLISHING, function (stream) {
...
}),publish();

2) After doing the suggested change (in above post by Max) on client side Streaming quality is good but stored video (s3) is still blur.

Please provide video samples or point to the folder where samples located.

3) Voice lagging in streaming video (voice and video sync up is not up to the mark) and some time it drops the voice from recorded video

See TCP switch recommendation above.

4) Need to enable full screen on the playing video (player) (Android Chrome browser)

You are free to modify default player example or use underling API.

Player

https://demo.flashphoner.com/client2/examples/demo/streaming/player/player.js

The same player is located on your server WCS_HOME/client2/examples/demo/streaming/player
You can copy this example to your web server for development.

5) Need to break video in parts instead of saving it as 1 big mp4 file in s3

You can setup recording rotation.
Example:

record_rotation=50M

6) We need compression while storing video on s3

WCS server does not support upload to S3 directly. So if you are using a custom script, you would need to implement compression in your script.
However, video compression is a CPU intensive operation and it will overload CPU and affect performance of the server.

7) Need the statistics like how many concurrent broadcast can run at a time with how many users (with EC2 c5.large instance)
8) Need the statistics like how many concurrent webinar can run at a time with how many users (with EC2 c5.large instance)

Statistics page is http://host:8081?action=stat
You can add this statistic into a monitoring tool like Zabbix or Prometheus

Load and resource usage information - Web Call Server 5.2 - EN - Flashphoner Documentation

docs.flashphoner.com

Additional information. We found memory misconfiguration.

WCS_HOME/conf/gc-core.log2020-06-12_06-06.log

288503.298: [Full GC (System.gc())  16740K->13808K(51396K), 0.0532471 secs]
289014.533: [GC (Allocation Failure)  28144K->13808K(51396K), 0.0016393 secs]
289499.556: [GC (Allocation Failure)  28144K->13901K(51396K), 0.0017616 secs]
289995.696: [GC (Allocation Failure)  28236K->13941K(51396K), 0.0018408 secs]
290500.631: [GC (Allocation Failure)  28277K->13902K(51396K), 0.0016834 secs]
290989.795: [GC (Allocation Failure)  28238K->13903K(51396K), 0.0016703 secs]
291497.464: [GC (Allocation Failure)  28239K->13829K(51396K), 0.0018166 secs]
291975.628: [GC (Allocation Failure)  28165K->13915K(51396K), 0.0017562 secs]
292103.352: [Full GC (System.gc())  18039K->13830K(51396K), 0.0553841 secs]
292614.564: [GC (Allocation Failure)  28166K->13831K(51396K), 0.0015919 secs]
293102.949: [GC (Allocation Failure)  28167K->13922K(51396K), 0.0016985 secs]
293598.541: [GC (Allocation Failure)  28242K->13910K(51396K), 0.0016993 secs]
294014.392: [GC (Allocation Failure)  28246K->13929K(51396K), 0.0018223 secs]
294504.644: [GC (Allocation Failure)  28265K->13930K(51396K), 0.0017617 secs]
295015.072: [GC (Allocation Failure)  28266K->13927K(51396K), 0.0017266 secs]
295504.071: [GC (Allocation Failure)  28263K->13929K(51396K), 0.0017740 secs]
295703.409: [Full GC (System.gc())  19647K->13770K(51396K), 0.0541482 secs]
296215.248: [GC (Allocation Failure)  28106K->13770K(51396K), 0.0016051 secs]
296694.373: [GC (Allocation Failure)  28098K->13881K(51396K), 0.0018602 secs]
297200.385: [GC (Allocation Failure)  28217K->13797K(51396K), 0.0016992 secs]
297687.513: [GC (Allocation Failure)  28133K->13881K(51396K), 0.0017886 secs]
298172.530: [GC (Allocation Failure)  28217K->13874K(51396K), 0.0017232 secs]
298683.211: [GC (Allocation Failure)  28210K->13798K(51396K), 0.0017337 secs]

Not enough heap memory. This might lead to a jitter or choppy video.

Solution:
1. Increase RAM.
2. Tune heap memory.
3. Install Java 12 and configure ZGC.
https://docs.flashphoner.com/display/WCS52EN/Before+moving+to+production (steps 11, 12)

 

[jyotipatidar]

Hi Max,
We tried the suggested solutions
1) Some times recording does not work
name: streamName,
display: localVideo,
cacheLocalResources: true,
receiveVideo: false,
receiveAudio: false,
transport: "TCP"

It gives Error - FAILED Failed by ICE timeout
Not able to implement this change yet and we are still stuck at video saving issue

2) Blur issue - Recorded Video uploaded @ /home/ec2-user/video
Please check and suggest the solution for this blur issue

On Memory recommendations we did 2 and 3 steps but
while configure ZGC in step 8 we are getting error on EC2 linux machine
systemctl enable rc-local.service
bash: systemctl: command not found
We checked and found that EC2 linux does not have this pkg installed

Thanks for your quick response !

 

[Max]

It gives Error - FAILED Failed by ICE timeout

Please check if media ports 31001-32000/TCP are opened in instance security group.

2) Blur issue - Recorded Video uploaded @ /home/ec2-user/video
Please check and suggest the solution for this blur issue

You are sharing screen from Chrome browsers without extension, in this case picture resolution constraints are not applied. So you should raise publishing bitrate to values suitable for FullHD:

    constraints: {
           video: {
                 minBitrate: 1500
                 maxBitrate: 2500
           }
    }

On Memory recommendations we did 2 and 3 steps but
while configure ZGC in step 8 we are getting error on EC2 linux machine

You can skip step 8, it is not necessary unless you prepare to high load. Please do not use hugepages.
Also, 700m is extremely low for Java heap

-Xmx700m -Xms700m

You must increase RAM on server at least to 4 Gb, RAM tuning is useless without it.

 

[jyotipatidar]

1) Now TCP related change is working, we will observe if it solves the video saving issue for a day or 2 - will update you
2) Which extension we need to install ? and is there anything we can do to make the extension mandatory before starting the streaming from client side ? with bitrate change it looks fine but consumes a lot of ntw.
3) 700 mb is our dev server. In prod we have 4 GB RAM - that is where we are testing right now. Below stats is for your ref. I hope this is fine
core_heap_memory_used=685655672
core_java_committedMemory=4392517632
core_java_threads=58
core_java_freePhysicalMemorySize=977125376
core_java_arch=amd64
core_java_availableProcessors=2
core_java_freeSwapSpaceSize=0
core_java_maxFileDescriptorCount=20000
core_java_open_file_descriptors=146
core_java_cpu_usage=0.02
core_java_totalPhysicalMemorySize=3887206400
core_java_totalSwapSpaceSize=0
core_java_uptime=1594388
core_java_version=12.0.2
Please suggest if any further optimization needed.

4) You can skip step 8, it is not necessary unless you prepare to high load
We might need this as we are expecting the load to go up.

5) On the server stats page I see this
ports_media_free=499
ports_media_busy=2
When there is 1 streaming going on with 1 participant it consumes 2 ports - does it mean it can support only 249 users ?
Can we increase this ? if so will there be any impact ?

6) Would you suggest to upgrade the version for these 2 for better performance ? if so please share the version number.
wcs_version=5.2.639
wcs_client_version=0.5.28.2753

Thanks !

 

[Max]

Which extension we need to install ? and is there anything we can do to make the extension mandatory before starting the streaming from client side ?

You should customize this extension to use with your domain. Please read this step by step doc
Note that you get lower quality while scaling desktop to lower resolution, so this will always be the comromise between quality and channel bandwidth.

700 mb is our dev server. In prod we have 4 GB RAM - that is where we are testing right now.

This may (and probably will) be not enough for you in production. Please read server capacity recommendations for common use cases here

We might need this as we are expecting the load to go up.

In AWS instance, use chkconfig instead of systemctl

chkconfig rc-local on

Can we increase this ? if so will there be any impact ?

Use media_port_from and media_port_to parameters to increase media ports range, for example

media_port_from=20001
media_port_to=40000

Please also read here how to prepare yor server to production use.

6) Would you suggest to upgrade the version for these 2 for better performance?

Seems it is not necessary for your case (screen sharing broadcast).

 

[jyotipatidar]

Thanks for plenty of information Max.
Major issue of saving the recording is still exists after switching to TCP.
Please provide a solution to resolve this issue on priority.

Attaching the image and logs for your reference.
After a certain time video files stucks at 0 kb size and it just stops saving. In the image I have marked the video's which failed to save.

Let me know if you need any other log file.

 

[Max]

Please collect a report as described here, including publishing client debug logs and send using this link

 

[jyotipatidar]

I have sent

 

[jyotipatidar]

Also
record_rotation=50M

is throwing exception. I have shared log file again
-----Errors info-----
java.nio.file.NoSuchFileException=1
org.apache.http.NoHttpResponseException=1
java.io.FileNotFoundException=1

 

[Max]

The exceptions are not related to recording (they were as well present in the other logs)
- FileNotFoundException: /usr/local/FlashphonerWebCallServer/conf/apps/click-to-call/accounts.xml
The file location is conf/click-to-call/accounts.xml - create in conf dir apps dir and move click-to-call/accounts.xml to conf/apps
- NoSuchFileException: /etc/hostname - specify the name of the server in /etc/host
(there is no NoHttpResponseException in the logs)

Which streams from the sent logs were not recorded?
There are still WARN ileRecordAudioBuffer "Full buffer and no video packets, dropping packets" in server log and WARN BitstreamNormalizer "Timing problem" in client log, that looks like network channel quality issue.
Do streams which are not recorded play normally?

 

[jyotipatidar]

Hi,
1) Any update on saving issue ? I had shared the dump yesterday
2) On record_rotation=50M , well we are breaking the video in 2 MB (record_rotation=2M) and for the first video it works fine but from second video after 2 MB it get's stuck (live steaming as well as saving into records folder)
Do streams which are not recorded play normally? - No it get's stuck after implementing the record_rotation=2M

 

[Max]

1) Any update on saving issue ? I had shared the dump yesterday

It still seems like channel quality/bandwidth issue. Please try to use lower resolution and bitrate (for screen sharing, implement the extension as we wrote above), relocate server instance close to publishing client, do not use 3G mobile networks.
You can test publisher channel using Media Devices example as described on this page:
1. Set the following parameters in flashphoner.properties file

inbound_video_rate_stat_send_interval=1 
outbound_video_rate_stat_send_interval=1

restart WCS
2. Publish 720p or 1080p stream with bitrate 1500-2500 kbps (as seen in client logs) from Media Devices example
3. You should see publisher channel state and bitrate graphs.

 

[jyotipatidar]

Hi,
relocate server instance close to publishing client - can we configure cloud front for this if so please share the step link

 

[Max]

Good day.

relocate server instance close to publishing client - can we configure cloud front for this if so please share the step link

You can download latest WebSDK build, unpack to your own webserver, embed flashphoner.js to your application code and use. Please read this article for ctep by step embedding instructions.
Note that stream is always published from browser, not from webserver where frontend page is deployed, so cloud front does not help to solve channel issues between browser and server.

 

[Max]

In WCS v. 5.2.668, an issue with audio quality in webm recording of VP8+Opus stream has been fixed (WCS-2719).