Flashphoner server logs when using media pipe selfie segmentation

[KaranBhansali91]

Hi,

Currently we are using mediapipe selfie segmentation for virtual background feature, the segmentation works with static images on live video , but the live video stucks when an image has remote url such as s3 url. Please find the screenshot of server log attached below.

 

[Max]

Good day.
Do you play the file as VOD from S3 storage and then add this to mixer? Or do you use a custom mixer layout implementation? Please clarify the case step by step to reproduce.
Please also note that that video may freeze when playing VOD from S3 storage if server and file storage are in different regions. Condsider to migrate the server closer to S3 storage.

 

[KaranBhansali91]

Hi,

No we are doing live streaming using canvas and we are using mixer to add participants. On the canvas we are using the virtual background with the help of https://google.github.io/mediapipe/solutions/selfie_segmentation library, which is than published to flashphoner media server. The image used for the virtual background is stored in s3 bucket.

 

[Max]

So, you publish video from canvas like Canvas Streaming or Stream Filter example. In this case, please chek if the stream published may be played from the server (before adding to mixer). Also, please check the stream metrics using /stream/find REST API:

POST /rest-api/stream/find HTTP/1.1
Host: wcs:8081
Content-Length: 57
Content-Type: application/json
 
{
    "name":"stream1",
    "published":true,
    "display":["metrics"]
}

If VIDEO_RATE metric value is 0 when you adding the remote image from s3, it means that browser stops sending video packets. For canvas streaming to work, the browser thread capturing an image from canvas must be on foreground. For example, if you switch to another browsers tab, canvas streaming will freeze.
So please check how do you downloading image from s3 and drawing on canvas. If download process goes to foreground and canvas drawing goes to background, it may be a reson for stream to freeze.
Another possible reason is channel bandwidth, this may be not enough to download remote image and to publish stream simultaneously. In this case, use the lower canvas resolution.
The last, but not least, please check clients PC CPU load. CPU may be not enough to replace the image background and capture stream from canvas simultaneously.