2 seconds freeze in stream

Dani

Member
many of my streams has lags (stream freeze for 2 seconds than continues) .
Is there a way to improve this ? what would be optimize settings to avoid these froze moments ?

server cpu is less than 10% (load average 0.9, 1.19, 1.33)

I've connected to the stream through several places and it freezes on the same time for all "players"
 
Last edited:

Max

Administrator
Staff member
Looks like possible memory issue.

Make sure your server has enough RAM and tuned as -Xmx16g -Xms16g

Here 16g is a half of RAM 32g

See also heap memory tuning

Check logs/gc-core.log
If these logs write too many messages like "Final remark" , it may be cause of freeze.

Please describe how do you test streams and provide SSH access. We will check on your server.
You can provide SSH using this private form.

See also production tuning guide
 

Dani

Member
java -Xmx16G -Xms16G -XX:+UseConcMarkSweepGC -XX:NewSize=1024m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=50999 -Djava.rmi.server.hostname=localhost -XX:ErrorFile=/usr/local/FlashphonerWebCallServer/logs/error%p.lo

THe server is open now, I've sent you the details. let me know if you can log in in the next hour.
 

Dani

Member
2021-01-21T09:04:41.041-0500: 2897.461: [GC (Allocation Failure) 2021-01-21T09:04:41.041-0500: 2897.461: [ParNew: 1088994K->44807K(1150208K), 0.0546900 secs] 1170951K->130591K(16649472K), 0.0551144 secs] [Times: user=0.49 sys=0.25, real=0.06 secs]
2021-01-21T09:05:15.752-0500: 2932.172: [GC (Allocation Failure) 2021-01-21T09:05:15.753-0500: 2932.173: [ParNew: 1067271K->41875K(1150208K), 0.0583821 secs] 1153055K->137048K(16649472K), 0.0588164 secs] [Times: user=0.49 sys=0.25, real=0.06 secs]
2021-01-21T09:05:51.392-0500: 2967.812: [GC (Allocation Failure) 2021-01-21T09:05:51.393-0500: 2967.812: [ParNew: 1064339K->32088K(1150208K), 0.0553944 secs] 1159512K->129388K(16649472K), 0.0558670 secs] [Times: user=0.41 sys=0.33, real=0.05 secs]

this is gc-core.log
 

Dani

Member
[root@cyber-net ~]# free -m
total used free shared buff/cache available
Mem: 32009 6844 369 1454 24795 23292
Swap: 2111 23 2088
 

Dani

Member
parameters from the client stream:
  1. cacheLocalResources: false
  2. constraints: {audio: false, video: true}
  3. display: div#DIVremoteVideo.display.center
  4. flashShowFullScreenButton: true
  5. mediaProvider: "WebRTC"
  6. name: "show49pmNN90yuAnV8210121170530867"
  7. __proto__: Object
 

Dani

Member
the stream is also freezing if I connect to it on your code:
which means the problem is on the publisher or the server.
if I publish the stream using your sample it works correct.
 

Dani

Member
one more input - it looks like the problem starts when we send the stream to Wowza server for recording.
This wasn't an issue in the old Flashphoner version we had, looks like its a new issue from this update (last version was from Jul 2019)
** confirmed - shutting down wowza server solved the freezing issue - but now we can't record....
 
Last edited:

Max

Administrator
Staff member
We checked your server.
Seems like streams are published from your frontend using VP8 video codec. When you republish such stream to Wowza via RTMP, video is transcoded to H264. So if you publishing more than 20 streams simultaneously, the CPU load may be too high depending on streams resolution (1 CPU core is required to transcode 2 streams 720p or 3 streams 480p), which can lead to freezes.
Also, you are using CMS garbage collector which also can affect server performance.
So we recomment the following:
1. Publish streams using H264 codec to avoid transcoding while republishing to RTMP server.
2. Update JDK to 12 or 14 and set up ZGC garbage collector as described here
If you're republishing streams for recording only, consider to record streams on WCS directly. Please read details here. Note that recording may be started and stopped on demand using REST API
 

Dani

Member
thanks for the help.
Although it's not a cpu issue (server never passed 10% utilization) , maybe wowza transcoding somehow tell the stream source to hold.... how do I change this:
Publish streams using H264 codec to avoid transcoding while republishing to RTMP server.

I will also try to apply the rest of your suggestion plus, for the long run move over to WCS recording now that it can do it on command and not from begging only.
 

Max

Administrator
Staff member
how do I change this:
Publish streams using H264 codec to avoid transcoding while republishing to RTMP server.
Please check if there is stripCodecs: "H264" options in client side code.
Also, set H264 codec as priotity in the following parameter:
Code:
codecs=opus,...,h264,vp8,...
h264 should be listed before vp8
I will also try to apply the rest of your suggestion plus, for the long run move over to WCS recording now that it can do it on command and not from begging only.
/stream/startRecording REST query may be applied to a stream published at any moment, not just in the beginning. You can even stop recording with /stream/stopRecording and then start recording again later.
Please note that you should know mediaSessionId for the stream to be recorded. This data may be obtained by /stream/find query by stream name:
Code:
POST /rest-api/stream/find HTTP/1.1
Host: wcs_address:8081
Content-Type: application/json
 
{
    "name":"stream1",
    "published":true
}
 

Dani

Member
I have changed the codec order on the server properties, I think that now the stream become very sensitive to movement. every small move - makes the entire frame pixelized
this is the current settings:
codecs =opus,alaw,ulaw,g729,speex16,g722,mpeg4-generic,telephone-event,h264,vp8,flv,mpv

running with stripCodec = "H264" and changing the order didn't solve the problem (but there is less freezing, it's just that the video is so bad we had to turn it back to vp8)
btw - the recording has no freezing issue - only the other stream players suffer from this.
 
Last edited:

Max

Administrator
Staff member
I have changed the codec order on the server properties, I think that now the stream become very sensitive to movement. every small move - makes the entire frame pixelized
This seems like VP8->H264 transcoding.
running with stripCodec = "H264" and changing the order didn't solve the problem
The stripCodecs option strips the codec from browsers SDP, so this codec is not used any more.
We recommend to remove the option from client code or set it to
Code:
stripCodecs: "VP8"
to exclude VP8 codec usage if you need H264 only.
Then, if stream quality is still bad, provide us SSH access to the server with publishing and playback ability for our engineers to check. Use this private form for credentials.
 
Last edited:
Top