Video distortion when changing RTCEAGLVideoView size

[andrew.n]

I use 2 RTCEAGLVideoViews, one to play the remote link and one for broadcasting. When the streamers are ready, the delegate method - (void)videoView: (RTCEAGLVideoView *)videoView didChangeVideoSize: (CGSize)size; is called and I set the videoView.layer.frame properly.

We have the following setup:
RTCEAGLVideoView 1 height constraint = UIScreen.current.bounds.height
RTCEAGLVideoView 1 width constraint = UIScreen.current.bounds.width
RTCEAGLVideoView 1 center is UIViewController.view.center

RTCEAGLVideoView 2 height constraint = 170
RTCEAGLVideoView 2 width contraint = 100
RTCEAGLVideoView 2 center is a fixed point at X = 200, Y = 200

Until now everything works fine, the issue is that when I change the following:
RTCEAGLVideoView 1 height constraint = 170
RTCEAGLVideoView 1 width constraint = 100
RTCEAGLVideoView 1 center is a fixed point at X = 200, Y = 200

RTCEAGLVideoView 2 height constraint = UIScreen.current.bounds.height
RTCEAGLVideoView 2 width constraint = UIScreen.current.bounds.width
RTCEAGLVideoView 2 center is UIViewController.view.center

Basically, I replace those 2 RTCEAGLVideoView dimensions and centered position.

The issue is that when I do this, the streaming is distorted and randomly the delegate method (void)videoView: (RTCEAGLVideoView *)videoView didChangeVideoSize: (CGSize)size; is called again and the streaming is fixed. But sometimes this method is not called even if I wait few minutes.
How can I force this reload and the point when I change the dimensions and positions for both RTCEAGLVideoViews.

Small note regarding the configuration:
let options = FPWCSApi2StreamOptions()
let constraints = FPWCSApi2MediaConstraints(audio: true, videoWidth: Int(UIScreen.main.bounds.width), videoHeight: Int(UIScreen.main.bounds.height))
options.constraints = constraints

 

[Max]

Good day.
Is the issue reproduced using TwoWayStreaming example? If not, please modify the TwoWayStreaming example code minimally to reproduce the issue and send us the code using this link.
This seems like WebRTC RTCEAGLVideoView issue. There is the ticket WCS-2794 to update WebRTC in iOS SDK, but there are also more priority tickets.

 

[andrew.n]

Good day.
Is the issue reproduced using TwoWayStreaming example? If not, please modify the TwoWayStreaming example code minimally to reproduce the issue and send us the code using this link.
This seems like WebRTC RTCEAGLVideoView issue. There is the ticket WCS-2794 to update WebRTC in iOS SDK, but there are also more priority tickets.

To reproduce the issue I will have to change the example by adding ViewController.xib ... Do you have any example by using XIBs/Storyboards instead of adding all outlets + constraints by code?

 

[Max]

Do you have any example by using XIBs/Storyboards instead of adding all outlets + constraints by code?

No, there are no such examples
If you cannot reproduce the issue by describing layout as code, maybe something wrong with XIB?

 

[andrew.n]

No, there are no such examples
If you cannot reproduce the issue by describing layout as code, maybe something wrong with XIB?

I sent the details about the implementation using the link you provided to me yesterday, I also included some screenshots with the flow.

 

[Max]

1) How can I make this flow happen without having to wait for the RTCEAGLVideoView to resize the content randomly after a few/more seconds? I would like to trigger this refresh by myself when I have to change the UI for both RTCEAGLVideoViews.

Seems like the delegate is called after key frame receiving. So the fiollowing server side settings should help:

periodic_fir_request = true
periodic_fir_request_interval = 2000
rtcp_pli_request_interval = 2000

This makes WCS to request key frame every 2 seconds. Therefore, delegate should be called no more than 2 seconds after size changing.

2) Can I replace the streaming sources for RTCEAGLVideoViews without waiting to make the connection again? (if I can replace the streaming sources for RTCEAGLVideoViews, this kind of refresh should be almost instant - is there any way to do this?)

You cannot replace RTCEAGLVideoView source without creating a new stream.
Please try to declare a two views for two streams and play with its visibility: hide one view and show another.
Please note those issues are from WebRTC library internals, and we cannot affect this directly without library code changing, but we have no plans to customize the library.

 

[andrew.n]

@Max We included those parameters in the configuration and looks like the refresh works better now but still is not fixing the whole flow.

As you can see the following debug logs, for the playURLVideoView the size delegate method is called multiple times.
The issue right now is that the play URL stream source is in landscape (video calling from Chrome on my MacBook) and as you can see, 1st time I receive a size where the width is less than the height, which represents a portrait size. That's why after the refresh the video view is filled and is not looking properly.
The 2nd time, as you can see, the width is bigger than the height, which is correct. But between the 1st and 2nd call, it takes about 36 seconds...

========= didChangeVideoSize =========
2020-08-13 2:13:42 PM +0000
fullScreenVideoView - playURL
videoView to refresh - playURLVideoView
<RTCEAGLVideoView: 0x106d39c50; frame = (192.055 415.656; 29.8891 64.6875); clipsToBounds = YES; autoresize = RM+BM; gestureRecognizers = <NSArray: 0x114793ac0>; layer = <CALayer: 0x106a08de0>>
(414.0, 896.0)
========= didChangeVideoSize =========
2020-08-13 2:14:18 PM +0000
fullScreenVideoView - playURL
videoView to refresh - playURLVideoView
<RTCEAGLVideoView: 0x106d39c50; frame = (0 331.562; 414 232.875); clipsToBounds = YES; autoresize = RM+BM; gestureRecognizers = <NSArray: 0x114793ac0>; layer = <CALayer: 0x106a08de0>>
(480.0, 270.0)



Also, yesterday I sent a message using the link you provided in this thread, including some screenshots and I didn't receive any confirmation that the message was received or any response at all.

 

[Max]

Also, yesterday I sent a message using the link you provided in this thread, including some screenshots and I didn't receive any confirmation that the message was received or any response at all.

Mail server itself does not send any confirmation receipt. But we answered to questions you ask in that message, see the post above.

As you can see the following debug logs, for the playURLVideoView the size delegate method is called multiple times.
The issue right now is that the play URL stream source is in landscape (video calling from Chrome on my MacBook) and as you can see, 1st time I receive a size where the width is less than the height, which represents a portrait size. That's why after the refresh the video view is filled and is not looking properly.
The 2nd time, as you can see, the width is bigger than the height, which is correct. But between the 1st and 2nd call, it takes about 36 seconds...

Please collect the report as described here including client debug logs and send us using this link, we will check keyframe interval.
If keyframes are sending evenly with interval set in server parameters, than problem is in WebRTC implementation library itself. We cannot affect delegate invokation by any way.
iOS SDK is just a wrapper above WebRTC library impelmenting websocket interaction between client and WCS server. This raw websocket protocol is opened and described here. So you can try to use any iOS AppRTC implementation and, if this works properly, integrate WCS raw websocket API to it.
Another option is to use WebSDK and develop PWA in Safari browser. In this case, all the pictire scaling is done by browser.

 

[Max]

We received your logs.
Keyframes are sent evenly, every 2 seconds.
How ever, we see periodic publisher resolution drop (for example, from 960x540 to 640x360 for publisher 1 and from 1080x1440 to 810x1080 for publisher 2). This usually means the channel between client and server quality or bandwidth issues.
Please try to use lower minimal bitrate (now it is set to 500000 bps) or lower resolutions. Also please check if freezes occurs while playing streams.
We also notices that some streams are failed due to audio RTP activity, so audio packets are not sent during 1 minute. Please check this.
So we recommend to stabilize the streaming and then check scaling issue again. If the issue still persists, it is probably WebRTC library issue as we wrote above.

 

[andrew.n]

We received your logs.
Keyframes are sent evenly, every 2 seconds.
How ever, we see periodic publisher resolution drop (for example, from 960x540 to 640x360 for publisher 1 and from 1080x1440 to 810x1080 for publisher 2). This usually means the channel between client and server quality or bandwidth issues.

I understand that this happens and that's fine.
The problem that I pointed out is that before I receive the quality size updates, I received the video size as potret mode, even the streaming was in landscape mode.

========= didChangeVideoSize =========
2020-08-13 2:13:42 PM +0000
fullScreenVideoView - playURL
videoView to refresh - playURLVideoView
<RTCEAGLVideoView: 0x106d39c50; frame = (192.055 415.656; 29.8891 64.6875); clipsToBounds = YES; autoresize = RM+BM; gestureRecognizers = <NSArray: 0x114793ac0>; layer = <CALayer: 0x106a08de0>>
(414.0, 896.0) << width < height (portrait mode)
========= didChangeVideoSize =========
2020-08-13 2:14:18 PM +0000
fullScreenVideoView - playURL
videoView to refresh - playURLVideoView
<RTCEAGLVideoView: 0x106d39c50; frame = (0 331.562; 414 232.875); clipsToBounds = YES; autoresize = RM+BM; gestureRecognizers = <NSArray: 0x114793ac0>; layer = <CALayer: 0x106a08de0>>
(480.0, 270.0) << width > height (landscape mode)

After this landscape update I received a few other sizes which are also in landscape mode but bigger size - which is ok. The issue is that why the first delegate call had the size in portrait mode?

 

[Max]

After this landscape update I received a few other sizes which are also in landscape mode but bigger size - which is ok. The issue is that why the first delegate call had the size in portrait mode?

Again, WebRTC traffic is received and displayed by WebRTC library. So this is library issue. We recommend to ask library developers for support in this case.
Please also note that browser WebRTC implementation seems less buggy than standalone one (which is linked with iOS SDK). So we recommend to develop web applications with WebSDK if native SDK capabilities is not enough.

 

[Max]

Good day.
We released iOS SDK 2.6.1:
- WebRTC library is actual, downloading from CocoaPods on every build
- Simple GPUImage usage example with beautifying filter
Please download iOS SDK library here https://flashphoner.com/downloads/builds/flashphoner_client/wcs-ios-sdk/2.6/WCS-iOS-SDK-2.6.1.tar.gz
Review the GPUImageDemo example description here https://docs.flashphoner.com/display/IOSSDK2EN/iOS+GPUImageDemo and source code here https://github.com/flashphoner/wcs-ios-sdk-samples/tree/2.6/WCSExample/GPUImageDemo
We still use RTCEAGLVideoView in our examples, but you can try to use RTCMTLVideoView as it supported by actual WebRTC library.

 

[andrew.n]

Thank you @Max for those great news!

 

[andrew.n]

@Max I changed the view to RTCMTLVideoView, but when the streaming stars, I see that the delegate method `func videoView(_ videoView: RTCVideoRenderer, didChangeVideoSize size: CGSize)` returns a static size of (360.0, 480.0), no matter what constraints I provide to
FPWCSApi2VideoConstraints. Also, I couldn't find a useful solution for this in the WebRTC docs. Can you tell me how can I improve the quality of the video? This video size is really awful for the last iPhones. We would like to have at least a full HD quality.

 

[Max]

Good day.
Seems like the lowest available resolution is always used for publishing. We reproduced the issue in TwoWayStreaming and MediaDevices examples and raised internal ticket WCS-2941. Will let you know results here.

 

[Max]

Good day.
We fixed the issue which allowed the lowest available resolution for publishing. Please update iOS SDK to build 2.6.2.