Add dynamic captions to a WebRTC stream

[kindco]

Hello @Max ,

We want to add dynamic captions to WebRTC stream videos. Caption means a short text somewhere in the bottom describing certain part of the video. Is there any possibility to achieve this with WCS?

Thanks!

 

[Max]

Good day.
You have a two options:
1. Create a transcoder for a certain stream using REST API /transcoder/startup, in this case you can add a picture as watermark

POST /rest-api/transcoder/startup HTTP/1.1
HOST: 192.168.1.101:8081
content-type: application/json
 
{
    "uri": "transcoder://tcode1",
    "remoteStreamName": "test",
    "localStreamName": "testT",
    "encoder": {
        "width": 640,
        "height": 480,
        "keyFrameInterval": 30,
        "fps": 30,
        "watermark": "/opt/media/Test.png"
    }
}

The picture to add should be placed to some folder in servers file system, /opt/media/ for example. PNG format is supported only.
2. Use custom decoded frame interceptor class. In this case you should implement Java class which will be called for each stream frame. You may receive the frame in YUV format, and modify it pixel by pixel, including text drawing anywhere on picture. Transcoding must be enabled for the stream (by playing it with explicit width and height setting for example) or for whole server by

disable_streaming_proxy=true

In both cases, stream transcoding is needed to decode stream pictures, modify and encode again, so be aware of CPU consumption. Usually, 1 CPU core is required to encode 2 720p streams. For 1 1080p stream, 2 cores may be required.

 

[kindco]

Hey @Max

Thank you for the answer!
So, both options are server-side only.
The first one won't work for us, because there can be only one watermark (we need dynamic captions, several per video, added on the fly depending on live stream content).
The second option can theoretically work, but, again, we need to know caption text on server side promptly, while stream is running, which sounds not trivial.

And just to be sure: is there any client-side possibility to add dynamic captions to the stream? I was thinking about canvas-based solution, when camera stream is first drawn on canvas, but this solution won't work in iOS Safari, as it relies on MediaRecorder which isn't supported there...

 

[Max]

And just to be sure: is there any client-side possibility to add dynamic captions to the stream? I was thinking about canvas-based solution, when camera stream is first drawn on canvas, but this solution won't work in iOS Safari, as it relies on MediaRecorder which isn't supported there...

Yes, you can capture stream from canvas, please read the technical details here and simple how-to guide here.
But this works in Chrome and Firefox only, so it will not work on iOS devices (because Chrome and Firefox on iOS doesn't support WebRTC).
Another option for iOS devices is to implement a native application and modify a picture using GPUImage library. iOS SDK support this in version 2.6. Please see GPUImageDemo example. But in this case you have to some native code in Objective C to implement custom capturer and application itself.