Video cut-off during recording

[Taylor]

Hello

We have been experiencing an odd issue lately where some of the videos recorded are cut-off halfway through the video recording.
For example, I started publishing a video stream and finished it 2 minutes afterwards but the video saved on the server was only 1 minute and 30 seconds long.

These video cut offs do not seem to be tied to any one device. We've had issues with it occurring on 'iMac - Safari', 'iMac - Chrome', 'Windows-Chrome' and 'iPad - Safari'. But they don't always consistently happen.

I've gone through the client and server logs and they all seem to imply that no issue occurred. With the example above, the status of 'Publishing' was sent and 2 minutes later the status of 'Unpublished' was sent with no other logs in the middle relating to the IP address or the stream name I was using.

One other detail about this was that while recording the example video above. 18 other people were in the middle of doing their own video recordings too.

Unfortunately this was only possible to replicate on the production servers. I have tried several times to replicate it on our UAT servers but to no avail. I can send some of the logs through via email but I will not be able to collect other logs from the production server, like the traffic dump.

Versions:
WCS version: 5.2.267-018743542d11cabedb2fdc2dd34e1fc68e676158
WebSDK version: 0.5.28.2753-155fd22204126ad44418ad42c7d26986f6d27cc6

Flashphoner Properties:
stream_record_policy_template={streamName}
wss.port=443
ws.port=80
periodic_fir_request=true
video_incoming_buffer_size=100

WCS-Core Properties:
-Xms12g -Xmx12g

Instance Type:
c5.2xlarge

This is all the information I have right now. I'm not sure what else I could potentially do, I will try to replicate this on UAT again. Any help on what the issue is would be greatly appreciated.

 

[Max]

Good day.
Please update to the latest build 5.2.555, there are many recording fixes since build you're using.
If the problem still persists on the latest build please collect a report as described here and send to support@flashphoner.com

 

[Taylor]

Good day Max

Thank you for responding quickly.

We would like to update to the latest build but the latest version on the AWS Marketplace is 5.2.389.
We are currently 4 months behind that version as well so we will upgrade to 5.2.389.

Since this is an issue on production I can only update as fast as is safely possible.

If you could give an idea as to when the latest version will be up on the AWS Marketplace that would be great as it would very much reduce risks and make it easier to update.

 

[Max]

when the latest version will be up on the AWS Marketplace

Expected this month, after some fixes for transcoding.

 

[Taylor]

Hi Max

I've managed to re-create the video cut-off on UAT, using WCS version 5.2.389 (latest AWS version).

The way I recreated it was to have many devices concurrently recording their videos to the same server (in this I had 18 devices at the same time).

I decided to then change the Amazon EC2 Instance Type to see if we could alter of fix this issue.
We started on t3.medium, then switched to t3.large, then to c5d.xlarge.
The results showed that the cut-offs happened at the same time for all the devices across the different instance types.

Other Information:
While I tested on 5.2.389, I did download and install 5.2.555 to test with that but I decided to rollback due to an issue in iPhone Safari. In it the iPhone decided to go full-screen when I started the video recording and when exiting full screen the video element on the page freezes on the last frame (while the recording/streaming continues in the background).

Another issue found, on both WCS 5.2.389 and 5.2.555 was that when testing with Safari on iMac, the camera and microphone permissions prompt would not display. I've tried restarting Safari, clearing cache and website data, resetting permissions in Safari->Preferences->Website, and I still cannot get permissions to be properly prompted or accepted. To note this only happens for Safari in iMac, I've tried it on iOS and MacBook and they don't seem to have this issue.

 

[Max]

Good day.

While I tested on 5.2.389, I did download and install 5.2.555 to test with that but I decided to rollback due to an issue in iPhone Safari. In it the iPhone decided to go full-screen when I started the video recording and when exiting full screen the video element on the page freezes on the last frame (while the recording/streaming continues in the background).

Please clarify:
- what iOS version do you test?
- what example application do you test: Stream Recording (but it has no fullscreen abilities), Player, Embed Player, etc? If you've tested your custom code, please provide us the minimal example to reproduce the issue (please send code example to support@flashphoner.com

Another issue found, on both WCS 5.2.389 and 5.2.555 was that when testing with Safari on iMac, the camera and microphone permissions prompt would not display.

Please clarify:
- what MacOS version and Safari version do you test?
- what example application do you test?

 

[Taylor]

Hey Max

I've just sent an error report regarding the video-cut off issue to your support team.

While I tested on 5.2.389, I did download and install 5.2.555 to test with that but I decided to rollback due to an issue in iPhone Safari. In it the iPhone decided to go full-screen when I started the video recording and when exiting full screen the video element on the page freezes on the last frame (while the recording/streaming continues in the background).

What iOS version do you test?
I used iOS 13.3.1

What example application do you test?
I used custom code to create this issue. I forgot to include in the post above that I tried it with the Stream Recording demo page, but it didn't do the full screen bug and instead worked completely fine.
I would like to test this out a bit more on other demo pages and inspect my custom code before handing it over to you, but I cannot do that right now due to a lack of time and resource.
When I do, I will make a post about it with greater detail and, if need be, send you through the required information to recreate it.

Another issue found, on both WCS 5.2.389 and 5.2.555 was that when testing with Safari on iMac, the camera and microphone permissions prompt would not display.

What MacOS version and Safari version do you test?
The MacOS is 10.15.3 Catalina and the Safari version is 13.0.5 (15608.5.11)
Just to note, this only happens on iMac device. Safari on MacBook was tested and that worked fine..

What example application do you test?
Again this is custom code. I am unable to confirm if this happens on the demo pages due to it getting an 'OverconstrainedError' when trying to start (I made a post about here: https://forum.flashphoner.com/threa...ror-on-specific-version-of-imac-safari.12133/).
If I have enough time today I would try patching the OverconstrainedError and then seeing if I can replicate the issue.


I'm not too fussed about the two above issues (iOS Fullscreen & iMac No Permission Prompt), I think it's only fair I collate more data about that first. I merely mentioned them to explain what testing results we got alongside the video cut-off.

But the video cut-off is something I would really like to be fixed, and so far I have not been able to find a solution.

If you can help us fix this that would be super awesome.

Cheers

 

[Max]

Good day.

I've just sent an error report regarding the video-cut off issue to your support team.

We asked you to grant access to download report files
Anyway, we strongly recommend you to update at least your staging server to latest build 5.2.556 and try to reproduce the issue. You can update WCS in AWS instance manually as described here.

When I do, I will make a post about it with greater detail and, if need be, send you through the required information to recreate it.

Please clarify, do you switch local video (stream published) to fullscreen or remote video (stream playing)?
Also please check if you use playFirstVideo() function call in iOS Safari before puiblishing/playing video?

 

[Taylor]

Hey Max

I've granted permissions this morning and sent an email back at the same time so you should have access to the report files.

Anyway, we strongly recommend you to update at least your staging server to latest build 5.2.556 and try to reproduce the issue.

This may have to wait a bit as testing it requires resources which I'm not sure I have right now. I hope to be able to test it with the latest server version within a week but I cannot guarantee that I'm afraid.
I am able to test the other issues that I mentioned above (iOS Fullscreen & iMac No Permission Prompt). I should be able to get back by tomorrow or the day after with more information, and on the latest WCS build.

Please clarify, do you switch local video (stream published) to fullscreen or remote video (stream playing)?

This happens upon stream publishing and having the local video element on the page switch to fullscreen by itself.
However this happens in my custom code though. When I tried it on the WCS 'Stream Recording' demo page, I couldn't recreate the issue.
To clarify, this happened on WCS 5.2.555. I didn't have it happen on WCS 5.2.389. I have yet to test WCS 5.2.556 but I will be doing that now.

Also please check if you use playFirstVideo() function call in iOS Safari before puiblishing/playing video?

Yes I do call that.
I cannot remember exactly why, but it was added to originally fix a Safari issue at the time.

 

[Max]

Good day.
We reproduced the several recording issue and raised the ticket WCS-2599. We'll let you know results here.

I am able to test the other issues that I mentioned above (iOS Fullscreen & iMac No Permission Prompt). I should be able to get back by tomorrow or the day after with more information, and on the latest WCS build.

We'll wait for more information from you. If the iOS and MacOS issues cannot be reproduced on our example applications, please provide us the minimal custom code example to reproduce the issue (no frameworks, pure JavaScript).

 

[Taylor]

Hey Max

Sorry for the late reply. I did however have success in replicating and understanding why these two issues happened.

iOS Fullscreen:
The reason this happened is that my implementation of 'playFirstVideo()' came from an old demo page of WCS (5.2.108) where I didn't use the 'PRELOADER_URL' or had it handle Promise callbacks correctly.
Once I updated it to use the same format as the Stream Recorder demo page (see below), it worked fine.

if (Browser.isSafariWebRTC()) {
            Flashphoner.playFirstVideo(localVideo, true, PRELOADER_URL).then(function() {
                startRecording();
            });
            return;
        }
        startRecording();

It didn't seem to have any issues until WCS 5.2.555, but once I upgrade to it, it then required the PRELOADER_URL. That was the key element that fixed it for iOS.

iMac No Permission Prompt:
So it turned out this was the 'OverconstrainedError' issue again that I've had before, and I had to fix it in my code in two places.

Firstly the method session.createStream() required to add the parameter 'disableConstraintsNormalization' and have it set to true.
Secondly the method Flashphoner.getMediaAccess() required the 'disableConstraintsNormalization' parameter to be set to true.

i didn't discover this immediately because my code calls 'getMediaAccess()' to determine if permissions were accepted or not, but since 'OverconstrainedError' was being triggered it went to the code that handled the denied permissions for Camera and Microphone.


So to summarise, there were faults and outdated methods used in my code which are fixed now. Sorry for the hassle and I hope the video cut-off issue is coming along well.

Cheers

 

[Taylor]

Hey Max

May I please know when the latest Flashphoner version will be available on the AWS Marketplace, and also what the status is with the video cut-off issue?

Cheers

 

[Max]

Good day.
We plan to update WCS build in AWS marketplace after some mixer and CDN transcoding fixes. You can update WCS to latest build in active AWS instance by installing new build over current one as described here.
Recording issue (ticket WCS-2599) fix is already on testing stage and coming soon.

 

[Max]

Good day.
We fixed simultaneous recording issue in build 5.2.639. Please update and check.
Note that latest AWS marketplace build is 5.2.629, so you should update WCS in active AWS instance by installing new build over current one as described here.

 

[Taylor]

Hey Max

Thank you for the update. I am unfortunately not able to test it immediately as I need my team members to reliably test the simultaneous recording issue and they are currently busy.
I am unsure when we'll be ready to test but when we do I will post the results below.

Cheers

 

[Taylor]

Hey Max

We've just tested it on build 5.2.673 with 19 different streams connecting at the same time. We didn't experience any videos being cut-off and all the saved videos ended when we expected them to.

So I am happy to report that the issue has been fixed.

Thank you very much for your help.