Error Messages in the logs for Mixer

[Luca]

Hi

We recently started to experience some strange behaviour with our WCS server:

1. Users randomly getting disconnected from conference Rooms

2. User stream stopping after sometimes it has been added to the mixer(the stream stopping is inside the mixer, the origin stream is fine), this usually happens after I have injected a stream into the mixer via the REST API, but majority of the times works fine

In the logs I keep getting this error(loads of them)

17:02:23,638 ERROR mixer://15003_en_p_1000 - MIXER-AGENT-mixer://15003_en_p_1000-53796ee4-ffcb-4a4a-91f7-4f22e4dc431a TRANSCODER RESULT NULL

But the mixer is behaving perfectly fine, however we keep getting loads of these errors.

In terms of the disconnection, I randomly get this log when it happens, but nothing else about the nature of the failure(IPs and Stream names have been replaced as they contain sensitive information)

}

"maxBitrate" : 3000

"height" : 720,

"width" : 1280,

"video" : {

"audio" : true,

"constraints" : {

"origin" : "",

},

"name" : 15003

"custom" : {

"mediaProvider" : "WebRTC",

},

"muted" : false

"videoState" : {

},

"muted" : true

"audioState" : {

"mediaType" : "publish",

"createDate" : 1671038325732,

"cvoExtension" : false,

"transport" : "UDP",

"codecImpl" : "",

"audioBitrate" : 0,

"fps" : 0,

"gop" : 0,

"history" : false,

"quality" : 0,

"maxBitrate" : 3000,

"minBitrate" : 0,

"bitrate" : 0,

"height" : 720,

"width" : 1280,

"record" : false,

"description" : "Audio RTP activity",

"info" : "Failed by RTP activity",

"videoCodec" : "H264",

"audioCodec" : "opus",

"status" : "FAILED",

"hasAudio" : true,

"hasVideo" : true,

"published" : true,

"name" : "15003-fictitious-5c60",

"mediaSessionId" : "5c6014f1-7bd3-11ed-821e-f9f3a319c2d6",

"sessionId" : "/0.0.0.0.:58452/0.0.0.0:8443-9659845b-c813-4115-9332-9384ceeda3bd",

"appKey" : "roomApp",

"nodeId" : "Bvn9oqdgIkpJ0eJGebHMFv8JhqSmOXEL@0.0.0.0",

{

This is the Warning that usually precedes the disconnection

17:21:45,548 WARN D - VideoProcessor-15003-5c60-5c6014f1-7bd3-11ed-821e-f9f3a319c2d6 Drop video frame from mixer's buffer

It use to work perfectly fine until a couple of days ago, there have been no changes in our infrastructure, how can we investigate please?

Thanks

 

[Max]

Good day.

17:02:23,638 ERROR mixer://15003_en_p_1000 - MIXER-AGENT-mixer://15003_en_p_1000-53796ee4-ffcb-4a4a-91f7-4f22e4dc431a TRANSCODER RESULT NULL

This is normal behavior for mixer. The message means a frame cannot be encoded in native C code due to some internal reason, and exception is raised to Java code.

description" : "Audio RTP activity",

"info" : "Failed by RTP activity",

This means audio media packets stops in the stream for 60 seconds, and the stream was stopped due to inactivity. So you should check the stream metrics: VIDEO_P_FRAMES, VIDEO_K_FRAMES, VIDEO_SYNC, AUDIO_SYNC should monotonically grow, VIDEO_BITRATE and AUDIO_BITRATE should not be 0 and should change up and down slightly.

17:21:45,548 WARN D - VideoProcessor-15003-5c60-5c6014f1-7bd3-11ed-821e-f9f3a319c2d6 Drop video frame from mixer's buffer

This means video packets are dropped from video mixer input buffer. Usually, it happens for bad media packets (broken synchronization value, no keyframe in time longer than buffer depth etc)

It use to work perfectly fine until a couple of days ago, there have been no changes in our infrastructure, how can we investigate please?

If you did not update WCS or did not change the settings, seems the problem is with publishers channels quality/bandwidth, or datacenter last mile network issues. The following should help:
- use TCP transport for such publishers
- reduce video resolution/bitrate for such publishers
- increase mixer input buffer

mixer_in_buffering_ms=600

- if using WCS build 5.2.1395 or later, enable strict jitter buffer

use_strict_jitter_buffer=true
jitter_buffer_capacity=30

 

[Luca]

Thank you

I have tried your suggestions but not improvements so far

I noticed that when I add my stream to the mixer the upload bandwidth increases quite a lot

I.e. if I do not add the stream it maintains around 400Kbps

If I add my stream to the mixer it shoots up to 1.8Mbps and depending on framerate it might go up to 3Mbps

Is this normal?

Also I am in the UK and the WCS server is in Switzerland in a datacentre with 1GBPS ISP that we use for production with 1000's of people using our services per hour, as you would understand we cannot install a WCS server close to each person using our service, what would be the best approach here?

Thanks

 

[Max]

I noticed that when I add my stream to the mixer the upload bandwidth increases quite a lot

I.e. if I do not add the stream it maintains around 400Kbps

If I add my stream to the mixer it shoots up to 1.8Mbps and depending on framerate it might go up to 3Mbps

Is this normal?

Please clarify how do you test? For example, if using Media Devices you can monitor a publishing bitrate and playback bitrate. The publishing bitrate consumer upload bandwidth, the playback bitrate consumes download one.
Also if you feeding a mixer with your stream, mixer tries to adjust the outgoing stream bitrate according to mixer_video_bitrate_kbps parameter (2 Mbps by default). So if there is at least one stream in mixer, mixers outgoing stream bitrate will not be less than 2 Mbps.

Also I am in the UK and the WCS server is in Switzerland in a datacentre with 1GBPS ISP that we use for production with 1000's of people using our services per hour, as you would understand we cannot install a WCS server close to each person using our service, what would be the best approach here?

The approach is the same as we recommended above:
1. Enable Publisher and player channel quality control
2. For clients with BAD channels try consequently:
- switching to TCP transport
- using lower piublishing resolution/bitrate
In both cases, the clients stream needs to be republished again with a new parameters
Also, you should recommend to your customers to use a better channel: for example, 4G/LTE instead of 3G, WiFi instead of mobile network.
A last but not least: 1Gbps is not so much for 1000 clients simultaneously. Suppose one client is publishing 1Mbps stream and plays 1 Mbps stream, in this case you may have maximum 400 clients on the server.
If there are a viewers in you case who only watching the streams, you can consider a CDN distributed by datacenters closer to viewers.