Mixer PiP flickering

[Petro O. Bochan]

Hello, I have just updated to 1440 and noticed that when a mixer is in PiP mode - there is some weird flickering. It wasn't like this before in previous versions - everything was smooth. The log shows this:

7:29:19,823 INFO D - VideoProcessor-af2/rec/nc65e/54ea9bf794514d05b261ed412d08ded5-ebea9f80-4d5e-11ed-8984-0b633ecb9f80 Add media process decoding
17:29:20,943 ERROR 4b8da42d6760d2161407 - MIXER-AGENT-mixer://m_27f8c8cb073f4b8da42d6760d2161407-006069d9-8515-448e-b573-92e63fc75f40 TRANSCODER RESULT NULL
17:29:25,212 ERROR 4b8da42d6760d2161407 - MIXER-AGENT-mixer://m_27f8c8cb073f4b8da42d6760d2161407-006069d9-8515-448e-b573-92e63fc75f40 TRANSCODER RESULT NULL
17:29:26,667 ERROR 4b8da42d6760d2161407 - MIXER-AGENT-mixer://m_27f8c8cb073f4b8da42d6760d2161407-006069d9-8515-448e-b573-92e63fc75f40 TRANSCODER RESULT NULL

Dropbox

www.dropbox.com

This is the mixer:

[
{
"localMediaSessionId": "15f4662a-d220-4254-946d-ba929574e270",
"localStreamName": "af2/rec/nc65e/live",
"uri": "mixer://m_27f8c8cb073f4b8da42d6760d2161407",
"status": "PROCESSED_LOCAL",
"hasAudio": true,
"hasVideo": true,
"record": false,
"transport": "UDP",
"mediaSessions": [
{
"localMediaSessionId": "e59d9dd0-4d5e-11ed-8c05-31af8167d4e8",
"audioLevel": 100,
"videoMuted": false,
"localStreamName": "af2/rec/nc65e/c0632250a7794967916524d647166cc3",
"boundStream": true
},
{
"localMediaSessionId": "ebea9f80-4d5e-11ed-8984-0b633ecb9f80",
"audioLevel": 100,
"videoMuted": false,
"localStreamName": "af2/rec/nc65e/54ea9bf794514d05b261ed412d08ded5",
"boundStream": true
}
],
"mixerLayoutClass": "com.flashphoner.mixerlayout.Af2Layout",
"mixerActivityTimerCoolOffPeriod": 1,
"mixerActivityTimerTimeout": -1,
"mixerAppName": "defaultApp",
"mixerAudioOpusFloatCoding": false,
"mixerAudioSilenceThreshold": -50.0,
"mixerAudioThreads": 2,
"mixerAutoScaleDesktop": true,
"mixerDebugMode": false,
"mixerDesktopAlign": "TOP",
"mixerDisplayStreamName": false,
"mixerFontSize": 20,
"mixerFontSizeAudioOnly": 40,
"mixerIdleTimeout": 3600000,
"mixerInBufferingMs": 500,
"mixerIncomingTimeRateLowerThreshold": 0.95,
"mixerIncomingTimeRateUpperThreshold": 1.05,
"mixerMcuAudio": false,
"mixerMcuVideo": false,
"mixerMcuMultithreadedMix": false,
"mixerMinimalFontSize": 1,
"mixerMcuMultithreadedDelivery": false,
"mixerOutBufferEnabled": true,
"mixerOutBufferInitialSize": 2000,
"mixerOutBufferStartSize": 1000,
"mixerOutBufferPollingTime": 100,
"mixerOutBufferMaxBufferingsAllowed": -1,
"mixerOutBufferOverflowRate": 0.15,
"mixerOutBufferOverflowDeviationWindow": 30000,
"mixerOutBufferOverflowAllowedDeviation": 1000,
"mixerShowSeparateAudioFrame": true,
"mixerTextAutoscale": true,
"mixerTextColour": "0xFFFFFF",
"mixerTextBulkWriteWithBuffer": true,
"mixerTextBulkWrite": true,
"mixerTextBackgroundOpacity": 100,
"mixerTextBackgroundColour": "0x2B2A2B",
"mixerTextPaddingLeft": 5,
"mixerVoiceActivitySwitchDelay": 0,
"mixerVoiceActivityFrameThickness": 6,
"mixerVoiceActivityFramePositionInner": false,
"mixerVoiceActivityColour": "0x00CC66",
"mixerVoiceActivity": false,
"mixerVideoWidth": 1280,
"mixerVideoThreads": 2,
"mixerVideoStableFpsThreshold": 15,
"mixerVideoQuality": 24,
"mixerVideoProfileLevel": "42c02a",
"mixerVideoLayoutDesktopKeyWord": "desktop",
"mixerVideoHeight": 720,
"mixerVideoGridLayoutPadding": 30,
"mixerVideoGridLayoutMiddlePadding": 10,
"mixerVideoFps": 30,
"mixerVideoDesktopLayoutPadding": 30,
"mixerVideoDesktopLayoutInlinePadding": 10,
"mixerVideoBufferLength": 1000,
"mixerVideoBitrateKbps": 1000,
"mixerUseSdpState": true,
"mixerType": "MULTI_THREADED_NATIVE",
"mixerThreadTimeoutMs": 33,
"mixerTextPaddingTop": 5,
"mixerTextPaddingRight": 4,
"mixerTextFont": "Serif",
"mixerTextPaddingBottom": 5,
"mixerTextDisplayRoom": true,
"mixerTextCutTop": 3,
"mixerRealtime": true,
"mixerPruneStreams": false,
"audioMixerOutputCodec": "opus",
"audioMixerOutputSampleRate": 48000,
"audioMixerOutputChannels": 2,
"audioMixerMaxDelay": 300,
"mixerAudioOnlyHeight": 360,
"mixerAudioOnlyWidth": 640,
"mixerTextOutsideFramePadding": 50,
"mixerTextAlign": "BOTTOM_LEFT",
"mixerVideoDesktopFullscreen": false,
"mixerLayoutDir": "",
"videoOutputCodec": "h264"
}
]

Thanks,
P

 

[Max]

Good day.
Please try to reproduce it with the default full screen desktop layout

mixer_video_desktop_fullscreen=true

and with the sample custom full screen layout. If the flickering is not reproduced, seems like this is your custom layout implementation issue, so review the code.

 

[Petro O. Bochan]

Thanks for a reply, here are the steps to reproduce this error without custom code as it is with the defaultApp:

http://localhost:8081/rest-api/mixer/startup

{
"localStreamName": "test2",
"uri": "mixer://m_abc2"
}

http://localhost:8081/rest-api/mixer/add

{
"uri": "mixer://m_abc2",
"remoteStreamName": "qqq"
}

10:53:59,223 ERROR lient-mixer://m_abc2 - MIXER-AGENT-mixer://m_abc2-dcddf175-693e-4824-bd87-2ae922fcdbf8 TRANSCODER RESULT NULL
10:54:01,735 ERROR lient-mixer://m_abc2 - MIXER-AGENT-mixer://m_abc2-dcddf175-693e-4824-bd87-2ae922fcdbf8 TRANSCODER RESULT NULL
10:54:11,821 ERROR lient-mixer://m_abc2 - MIXER-AGENT-mixer://m_abc2-dcddf175-693e-4824-bd87-2ae922fcdbf8 TRANSCODER RESULT NULL

 

[Max]

10:53:59,223 ERROR lient-mixer://m_abc2 - MIXER-AGENT-mixer://m_abc2-dcddf175-693e-4824-bd87-2ae922fcdbf8 TRANSCODER RESULT NULL

This is not an error. The message means mixer native code cannot encode the frame for some reason. In this case, the frame is dropped. Usually, this may be due to packet loss or frame corruption. You can try a strict jitter buffer to reduce this

use_strict_jitter_buffer=true
jitter_buffer_capacity=30

And that's not a reason for flickering. We can't reproduce flickering by this way:
1. Create a mixer with "mixerVideoDesktopFullscreen": true
2. Add 3 streams test1, test2, test1_desktop
3. Play the mixer stream
Speech indicator slightly flickers around the picture. If it is disabled, no any flicker at all.

 

[Petro O. Bochan]

Finally we found the culprit: for some reason "mixer_video_threads" was set to 2. Once we have set it to 8 - flickering was gone like never existed. We thought that the value of this property should have been set to the CPU threads value, and since we have been testing on a low-level machine to ensure the code behaves correctly on a more powerful machine - we have been experiencing hard time flickering.

With that, what is this property for and what should be its value for e.g. 8, 10, 20, or 24 core machine?

Thanks,
P

 

[Max]

With that, what is this property for and what should be its value for e.g. 8, 10, 20, or 24 core machine?

The mixer_video_threads sets how much threads WCS will use to encode video in mixer output stream. This value is 4 by default, and should not be less because encoder may fail to process all incoming data. Usually, 4 threads is enough for most cases unless you're feeding the mixer with a huge amount of participants (10 incoming streams and more).

 

[Petro O. Bochan]

Got it, thanks.