rtmp republishing artifacts replay issues

S

Stian Eide

Member
Hi

running version 5.2.567 we seen a lot of issues with artifacts on the republished stream, see attached image

We think this was not an issue on older versions but we´r not really sure when this was introduced, it also seems to help putting the vp8 codec behind h264 in the flashphoner.properties at least on republishing from IOS

We tried republishing to both a nimble and a wowza server with the same result, this issue seems to be on all our flashphoner servers
 

Attachments

Max

Max

Administrator
Staff member
Hello,

The behavior has not been reproduced with WebRTC stream from iOS Safari pushed as RTMP to Wowza.
Please clarify
- how stream is published (device, browser version, codec)
- how it is republished (rtmpUrl is specified when stream is published, or rest-api/push/startup request is used)
- is the issue reproducible if republish to localhost (i.e., the same WCS - rtmp://localhost:1935/live)
 
S

Stian Eide

Member
Hi Max
Published from both android, chrome v 80.0.3987.149 and ios 13.4
Code: 
codecs                   =opus,alaw,ulaw,g729,speex16,g722,mpeg4-generic,telephone-event,h264,vp8,flv,mpv
codecs_exclude_sip       =mpeg4-generic,flv,mpv
codecs_exclude_streaming =flv,telephone-event
codecs_exclude_sip_rtmp  =opus,g729,g722,mpeg4-generic,vp8,mpv
Republished using /rest-api/push/startup

I will test republishing to localhost and get back with results, and also check if i can have a consistent scenario with these artifacts you can test
 
Last edited:
S

Stian Eide

Member
HI Max

We still see this issue with artifacts across several installations, the video in the link is recorded locally (rtmp://localhost:1935/live) from a webrtc stream udp (we see the same with tcp) version on server 5.2.603

Device android, huwaei p10 chrome 81.0.4044.138 , codec vp8 , but we see it also from IOS devices
 
S

Stian Eide

Member
Not sure if it´s related but i tested to stream rtmp directly to see how the recording looked like, the rtmp stream from OBS disconnects periodically every 40-60 seconds , OBS log
19:00:55.280: ==== Streaming Start ===============================================
19:01:45.754: WriteN, RTMP send error 10053 (2621 bytes)
19:01:45.754: WriteN, RTMP send error 10053 (39 bytes)
19:01:45.754: WriteN, RTMP send error 10038 (42 bytes)
19:01:45.754: [rtmp stream: 'adv_stream'] Disconnected from rtmp://xxx:1935/live
19:01:45.754: Output 'adv_stream': stopping
19:01:45.754: Output 'adv_stream': Total frames output: 1488
19:01:45.754: Output 'adv_stream': Total drawn frames: 1522
19:01:45.754: Output 'adv_stream': Reconnecting in 10 seconds..

Also the recording from OBS looks like it have issues in the standard windows mediaplayer, in vlc playback looks ok

Streaming rtmp from larix broadcaster(android app) works fine, the republished video looks also as expected
 
Last edited:
Max

Max

Administrator
Staff member
Good day.
Stian Eide said:
the rtmp stream from OBS disconnects periodically every 40-60 seconds , OBS log
...
Also the recording from OBS looks like it have issues in the standard windows mediaplayer, in vlc playback looks ok
Click to expand...
This is the known OBS issue: you should set the following parameter in flashphoner.properties
Code: 
keep_alive.enabled=websocket,rtmfp
to prevent disconnection.
Also please set encoding profile to baseline and encoding preset to ultrafast
1589247654121.png

to prevent B-frames in RTMP stream if you will play the stream as WebRTC (most browsers do not support B-frames playback)
Stian Eide said:
Device android, huwaei p10 chrome 81.0.4044.138 , codec vp8 , but we see it also from IOS devices
Click to expand...
You are publishing VP8+Opus stream, then republishing it as H264+AAC. There is transcoding on the server, an it seems like CPU performance is not enough to do it, this lead to picture artifacts. So you should either publish H264+Opus (in this case only audio will be transcoded which is mush less resource consuming operation) or migrate to more powerful server assuming 1 CPU core per 2 720p published streams plus a couple of cores for other operations.
 
S

Stian Eide

Member
Thank you Max for the reply

I resized the server from 2 cpu´s to 4 without any difference, when removing the vp8 codec it works just fine, but we need to have the vp8 as fallback, also when changing the "priority" so that h264 is before vp8 the stream is not good at least from my huawei p10, it will not run without the vp8 codec. It seems to me that the transcoding when using vp8 has some issues on our server and that it's not related to lack of resources.

-----Native Resources-----
native_resources=139821058969488,NENC:H264/OPENH264,914;139821059312912,FFDecoderNative:VP8/FFMPEG,2162882
native_resources.audio_codecs=0
native_resources.audio_resamplers=0
native_resources.video_transcoders=0
native_resources.video_decoders=1
native_resources.video_encoders=1
native_resources.writers=0
-----Core Stats-----
core_heap_memory_used=553298608
core_java_committedMemory=4244844544
core_java_threads=93
core_java_freePhysicalMemorySize=6044786688
core_java_arch=amd64
core_java_availableProcessors=4
core_java_freeSwapSpaceSize=268431360
core_java_maxFileDescriptorCount=20000
core_java_open_file_descriptors=198
core_java_cpu_usage=5.46
core_java_totalPhysicalMemorySize=8201068544
core_java_totalSwapSpaceSize=268431360
core_java_uptime=129877
core_java_version=1.8.0_201
-----Call Stats-----
 
Max

Max

Administrator
Staff member
Stian Eide said:
I resized the server from 2 cpu´s to 4 without any difference
Click to expand...
Please clarify what resolution do you streaming and what resolution do you set for republishing? How much streams are published simultaneously?
Transcoding consumes a much of server resources, so resizing from 2 to 4 vCPUs is not enough. Please consider more powerful configuration. We also recommend to use hardware server, not virtual, for this purpose.
Another option is to use lower publishing resolution, 480p for example.
 
S

Stian Eide

Member
Thanks Max, our test is only 1 simultaneous stream 480x640 so nothing heavy, i can change it to an instance with 8 dedicated CPU´s and see if that changes anything

-----Transcoding info-----
transcoding_video_decoding_resolutions=480x640/1
transcoding_video_decoding_average_time=480x640/1.0
transcoding_video_decoding_max_time=480x640/1
transcoding_video_decoding_average_queue_size=480x640/0.0
transcoding_video_decoding_max_queue_size=480x640/0
transcoding_video_decoding_load=8601600
transcoding_video_encoding_resolutions=0x0/2
transcoding_video_encoding_average_time=0x0/1.5
transcoding_video_encoding_max_time=0x0/3
transcoding_video_encoding_average_queue_size=0x0/0.0
transcoding_video_encoding_max_queue_size=0x0/0
transcoding_video_encoding_load=0

top.PNG
 
Max

Max

Administrator
Staff member
Good day.
We tried to reproduce the issue on VPS with 4 vCPU (ESXi host):
- stream 480x640 (vertical) is publishing from Media Device example (Chrome 81, Nokia 5 Android 9) with stripCodecs: "H264"
- stream is republishing to localhost using REST API /push/startup (video transcoding)
- RTMP stream is recording on server, is playing with VLC as RTMP and in Chrome browser as WebRTC (audio transcoding from AAC to Opus)
No transcoding artifacts are reproduced. If we didn't set bitrate constraints, there was bitrate drop artifacts in published stream, those artifacts also can be observed in RTMP stream.
So please check the following:
1. Try to set bitrate constraints on client side
Code: 
constraints.video.minBitrate = 500;
constraints.video.maxBitrate = 700;
or on server side
Code: 
webrtc_cc_min_bitrate=500000
webrtc_cc_max_bitrate=700000
and check if artifacts persist
2. Try to play and record WebRTC stream published from a phone. Check if artifacts are observed in published stream.
3. Check stream published from a phone parameters using REST API /stream/metrics. Are video packets loss in the metrics?
Also please collect a report as described here (we asked you some posts above) including client debug logs and traffic dump, then send it to support@flashphoner.com
SSH access to the server may also be useful, we will try to reproduce the isssue directly on server.
 
S

Stian Eide

Member
Thank you Max

I´v sent a report to the support email, we always have bitrate constraints on the server side

we use

Code: 
webrtc_cc_min_bitrate=500000
webrtc_cc_max_bitrate=3000000
I also tried with
Code: 
webrtc_cc_min_bitrate=500000
webrtc_cc_max_bitrate=700000
Without any changes in the artifacts issue on the republished rtmp stream, the "webrtc" stream is playing without any artifacts

There are some lost video packets in the metrics, i sent details on that in the email as well
 
Max

Max

Administrator
Staff member
We checked your logs. The metrics show:
1) 1% video losses per 2 minutes (40 frames of ~4000)
2) Long key frame interval (average 15 seconds)
3) High bitrate (1,6 Mbps for 480p is high)
Code: 
{
"VIDEO_SYNC":1589369291689,
"VIDEO_K_FRAMES":8,
"AUDIO_SYNC":0,
"VIDEO_NACK":40,
"AUDIO_RATE":0,
"AUDIO_LOST":0,
"VIDEO_LOST":40,
"VIDEO_CODEC":120,
"VIDEO_B_FRAMES":0,
"VIDEO_PLI":0,
"AUDIO_CODEC":0,
"VIDEO_RATE":1671648,
"VIDEO_WIDTH":480,
"VIDEO_HEIGHT":640,
"VIDEO_FPS":28,
"VIDEO_P_FRAMES":4353
}
So we recommend:
1) Check you channel to server quality (change network, use Wi-Fi instead of 4G, check if artifacts persists when publishing stream from desktop)
2) Enable periodic FIR request on server side for keyframe interval to be more even
Code: 
periodic_fir_request=true
4) Limit bitrate on client side using constraints
Code: 
    session.createStream({
        name: streamName,
        display: localVideo,
        cacheLocalResources: true,
        receiveVideo: false,
        receiveAudio: false,
        rtmpUrl: rtmpUrl,
        constraints: {video: {width: 640, height: 480, minBitrate: 500, maxBitrate: 700}}
    }).on(STREAM_STATUS.PUBLISHING, function(publishStream){
        ...
    }).publish();
If this does not help, please provide us SSH access to your server, we will check in place. Send credentials to support@flashphoner.com.
 
S

Stian Eide

Member
HI Max

Thanks for looking into this, i will try your recommendations and get back with the results but i´m quite sure we´r looking the wrong place based on

1 it only occurs when using vp8
2 It´s reproducible on demo.flashphoner.com( i sent some video´s to the support) i guess bitrate is limited there, 480p recommended encoding bitrate on youtube is 2.5-4 mbps
3 It happens on both wifi and 4g , speedtest on wifi is upload 50 mbps , latency 18ms, jitter 7ms no packet loss , on 4g+ it´s 17 Mbps 18ms latency, jitter 3ms no packet loss
4 It happens on multiple devices both android and IOS when using vp8 before h264
5 switching to tcp does not help


Does ffmepg use multiple cpu´s when transcoding? and what is the settings for that ?
 
S

Stian Eide

Member
Hi Max

Our first finding today is that
Code: 
periodic_fir_request=true
Seems to a really big difference, we will continue to do some more testing and get back with results

Thank you so far for the good support
 
Max

Max

Administrator
Staff member
Stian Eide said:
Does ffmepg use multiple cpu´s when transcoding? and what is the settings for that ?
Click to expand...
By default, OpenH264 encoder is used due to FFmpeg license limits. This encoder does not use multithreading.
Stian Eide said:
It´s reproducible on demo.flashphoner.com
Click to expand...
Unfortunately, we can't reproduce the issue on demo.flashphoner.com, that's why we suppose a channel problem.
If recommendations doesn't help, please provide us SSH access to your server, we will check in place.
 
S

Stian Eide

Member
Hi, i can see that in version 5.2.816 OPENH264 multithreaded encoding is supported, will it work by default or do we need to configure it ? And does it mean that it will use mulithreading when transcoding vp8 for republishing ?
 
Max

Max

Administrator
Staff member
Regarding VP8.
Yes. If you transcode VP8 to H.264, multithreading will be applied to H.264 encoder.
 
You must log in or register to reply here.
Top