Streams stop working restart of wcs needed

[stone]

running 5.2.1126

Video streams are intermittently failing and a restart of the wcs service is needed.

seeing errors like this in the log

14:41:48,951 ERROR RtspAgent - API-ASYNC-pool-13-thread-62 Stream failed
14:41:48,952 INFO RestClient - API-ASYNC-pool-13-thread-49 SEND REST OBJECT ==>
URL:http://localhost:8081/apps/EchoApp/StreamStatusEvent
OBJECT:
{
"nodeId" : "dhXrdMH4yHfngJXudpeDirwZxNyE9TkJ@10.10.28.17",
"appKey" : "defaultApp",
"sessionId" : "//:8443-cb97ce8f-e98b-40b1-9322-f38f703923fc",
"mediaSessionId" : "a0afc860-d70f-11ee-b64e-e7fbb7d9f35d",
"name" : "rtsp://admin:%23@8:554/unicast/c15/s2/live",
"published" : false,
"hasVideo" : true,
"hasAudio" : true,
"status" : "FAILED",
"audioCodec" : "PCMU",
"videoCodec" : "H264",
"info" : "Stream failed",


I also see a GC-core log file but nothing obvious in that file:

[2024-02-29T10:57:51.558+0000] Heap region size: 1M
[2024-02-29T10:57:51.601+0000] Using G1
[2024-02-29T10:57:51.602+0000] Heap address: 0x00000000c0000000, size: 1024 MB, Compressed Oops mode: 32-bit
[2024-02-29T10:57:51.602+0000] Mark closed archive regions in map: [0x00000000ffc00000, 0x00000000ffc6aff8]
[2024-02-29T10:57:51.603+0000] Mark open archive regions in map: [0x00000000ff800000, 0x00000000ff847ff8]
[2024-02-29T10:57:54.898+0000] GC(0) Pause Young (Normal) (G1 Evacuation Pause)
[2024-02-29T10:57:54.903+0000] GC(0) Using 18 workers of 18 for evacuation
[2024-02-29T10:57:54.921+0000] GC(0) Pre Evacuate Collection Set: 0.1ms
[2024-02-29T10:57:54.921+0000] GC(0) Evacuate Collection Set: 16.3ms
[2024-02-29T10:57:54.921+0000] GC(0) Post Evacuate Collection Set: 1.2ms
[2024-02-29T10:57:54.921+0000] GC(0) Other: 5.5ms
[2024-02-29T10:57:54.921+0000] GC(0) Eden regions: 51->0(44)
[2024-02-29T10:57:54.921+0000] GC(0) Survivor regions: 0->7(7)
[2024-02-29T10:57:54.921+0000] GC(0) Old regions: 2->7
[2024-02-29T10:57:54.921+0000] GC(0) Humongous regions: 2->0
[2024-02-29T10:57:54.921+0000] GC(0) Metaspace: 13361K(14080K)->13361K(14080K) NonClass: 11853K(12288K)->11853K(12288K) Class: 1507K(1792K)->1507K(1792K)
[2024-02-29T10:57:54.921+0000] GC(0) Pause Young (Normal) (G1 Evacuation Pause) 53M->12M(1024M) 23.266ms
[2024-02-29T10:57:54.921+0000] GC(0) User=0.16s Sys=0.07s Real=0.02s
[2024-02-29T10:57:56.080+0000] GC(1) Pause Young (Normal) (G1 Evacuation Pause)
[2024-02-29T10:57:56.080+0000] GC(1) Using 18 workers of 18 for evacuation
[2024-02-29T10:57:56.117+0000] GC(1) Pre Evacuate Collection Set: 0.0ms
[2024-02-29T10:57:56.117+0000] GC(1) Evacuate Collection Set: 35.5ms
[2024-02-29T10:57:56.117+0000] GC(1) Post Evacuate Collection Set: 1.3ms
[2024-02-29T10:57:56.117+0000] GC(1) Other: 0.4ms
[2024-02-29T10:57:56.117+0000] GC(1) Eden regions: 44->0(45)

 

[Max]

Hello

1. Gather report

sudo /usr/local/FlashphonerWebCallServer/tools/report.sh --sysinfo --conf --dump --tar

2. Send us via this form or directly to support@flashphoner.com if form is not reachable.
3. Our engineers will analyze logs and provide report back here in the thread.

 

[Max]

We checked the report. Seems like you've removed some files (jstack.log, Java heap dump) from the report. Also, you've replaced some info. Please, don't do this again.
The problem is lack of Java heap memory:

You're using a default Java heap size (1 Gb)

This may be not enough under a high load. The server has 74 Gb RAM, so we recommend to set at least 32 Gb for Java heap in /usr/local/FlashphonerWebCallServer/conf/wcs-core.properties file:

-Xmx32g
-Xms32g