2 seconds freeze in stream

[Dani]

many of my streams has lags (stream freeze for 2 seconds than continues) .
Is there a way to improve this ? what would be optimize settings to avoid these froze moments ?

server cpu is less than 10% (load average 0.9, 1.19, 1.33)

I've connected to the stream through several places and it freezes on the same time for all "players"

 

[Max]

Looks like possible memory issue.

Make sure your server has enough RAM and tuned as -Xmx16g -Xms16g

Here 16g is a half of RAM 32g

See also heap memory tuning

Memory management in Java - Web Call Server 5.2 - EN - Flashphoner Documentation

docs.flashphoner.com

Check logs/gc-core.log
If these logs write too many messages like "Final remark" , it may be cause of freeze.

Please describe how do you test streams and provide SSH access. We will check on your server.
You can provide SSH using this private form.

See also production tuning guide

 

[Dani]

java -Xmx16G -Xms16G -XX:+UseConcMarkSweepGC -XX:NewSize=1024m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=50999 -Djava.rmi.server.hostname=localhost -XX:ErrorFile=/usr/local/FlashphonerWebCallServer/logs/error%p.lo

THe server is open now, I've sent you the details. let me know if you can log in in the next hour.

 

[Dani]

2021-01-21T09:04:41.041-0500: 2897.461: [GC (Allocation Failure) 2021-01-21T09:04:41.041-0500: 2897.461: [ParNew: 1088994K->44807K(1150208K), 0.0546900 secs] 1170951K->130591K(16649472K), 0.0551144 secs] [Times: user=0.49 sys=0.25, real=0.06 secs]
2021-01-21T09:05:15.752-0500: 2932.172: [GC (Allocation Failure) 2021-01-21T09:05:15.753-0500: 2932.173: [ParNew: 1067271K->41875K(1150208K), 0.0583821 secs] 1153055K->137048K(16649472K), 0.0588164 secs] [Times: user=0.49 sys=0.25, real=0.06 secs]
2021-01-21T09:05:51.392-0500: 2967.812: [GC (Allocation Failure) 2021-01-21T09:05:51.393-0500: 2967.812: [ParNew: 1064339K->32088K(1150208K), 0.0553944 secs] 1159512K->129388K(16649472K), 0.0558670 secs] [Times: user=0.41 sys=0.33, real=0.05 secs]

this is gc-core.log

 

[Dani]

[root@cyber-net ~]# free -m
total used free shared buff/cache available
Mem: 32009 6844 369 1454 24795 23292
Swap: 2111 23 2088

 

[Dani]

parameters from the client stream:

1. cacheLocalResources: false
2. constraints: {audio: false, video: true}
3. display: div#DIVremoteVideo.display.center
4. flashShowFullScreenButton: true
5. mediaProvider: "WebRTC"
6. name: "show49pmNN90yuAnV8210121170530867"
7. __proto__: Object

 

[Dani]

the stream is also freezing if I connect to it on your code:

Two-way Streaming

which means the problem is on the publisher or the server.
if I publish the stream using your sample it works correct.

 

[Dani]

one more input - it looks like the problem starts when we send the stream to Wowza server for recording.
This wasn't an issue in the old Flashphoner version we had, looks like its a new issue from this update (last version was from Jul 2019)
** confirmed - shutting down wowza server solved the freezing issue - but now we can't record....

 

[Max]

We checked your server.
Seems like streams are published from your frontend using VP8 video codec. When you republish such stream to Wowza via RTMP, video is transcoded to H264. So if you publishing more than 20 streams simultaneously, the CPU load may be too high depending on streams resolution (1 CPU core is required to transcode 2 streams 720p or 3 streams 480p), which can lead to freezes.
Also, you are using CMS garbage collector which also can affect server performance.
So we recomment the following:
1. Publish streams using H264 codec to avoid transcoding while republishing to RTMP server.
2. Update JDK to 12 or 14 and set up ZGC garbage collector as described here
If you're republishing streams for recording only, consider to record streams on WCS directly. Please read details here. Note that recording may be started and stopped on demand using REST API

 

[Dani]

thanks for the help.
Although it's not a cpu issue (server never passed 10% utilization) , maybe wowza transcoding somehow tell the stream source to hold.... how do I change this:
Publish streams using H264 codec to avoid transcoding while republishing to RTMP server.
I will also try to apply the rest of your suggestion plus, for the long run move over to WCS recording now that it can do it on command and not from begging only.

 

[Max]

how do I change this:
Publish streams using H264 codec to avoid transcoding while republishing to RTMP server.

Please check if there is stripCodecs: "H264" options in client side code.
Also, set H264 codec as priotity in the following parameter:

codecs=opus,...,h264,vp8,...

h264 should be listed before vp8

I will also try to apply the rest of your suggestion plus, for the long run move over to WCS recording now that it can do it on command and not from begging only.

/stream/startRecording REST query may be applied to a stream published at any moment, not just in the beginning. You can even stop recording with /stream/stopRecording and then start recording again later.
Please note that you should know mediaSessionId for the stream to be recorded. This data may be obtained by /stream/find query by stream name:

POST /rest-api/stream/find HTTP/1.1
Host: wcs_address:8081
Content-Type: application/json
 
{
    "name":"stream1",
    "published":true
}

 

[Dani]

I have changed the codec order on the server properties, I think that now the stream become very sensitive to movement. every small move - makes the entire frame pixelized
this is the current settings:
codecs =opus,alaw,ulaw,g729,speex16,g722,mpeg4-generic,telephone-event,h264,vp8,flv,mpv

running with stripCodec = "H264" and changing the order didn't solve the problem (but there is less freezing, it's just that the video is so bad we had to turn it back to vp8)
btw - the recording has no freezing issue - only the other stream players suffer from this.

 

[Max]

I have changed the codec order on the server properties, I think that now the stream become very sensitive to movement. every small move - makes the entire frame pixelized

This seems like VP8->H264 transcoding.

running with stripCodec = "H264" and changing the order didn't solve the problem

The stripCodecs option strips the codec from browsers SDP, so this codec is not used any more.
We recommend to remove the option from client code or set it to

stripCodecs: "VP8"

to exclude VP8 codec usage if you need H264 only.
Then, if stream quality is still bad, provide us SSH access to the server with publishing and playback ability for our engineers to check. Use this private form for credentials.