Master Origin server is unresponsive

[manikandan]

Hi,

My stream server setup:
WCS Version: 5.2.699
Using Origin and edge server combination and one individual server as master origin which acts as source for other origin servers. This setup is working fine for almost more than 6months.

From yesterday, my master origin server seems to be unresponsive (while other origin servers working) and no streams getting published. No error logs generated in /logs/server_logs/flashphoner.log

On checking with other logs, I can see below errors in gc-core and dtls erros logs

My server has 4GB of RAM and configured 75% of memory in wc-core.properties

May I know what will be the issue?

 

[Max]

Good day.
Seems like your server crashed due to out of memory and/or out of disk space. Please check if disk space available

df -h

If not, clean up disk and then restart the server.

My server has 4GB of RAM and configured 75% of memory in wc-core.properties

This may be too low depending on server load. Considers to ugrade to at least 8 Gb (4Gb Java heap).
To work more stable under high load, we also recommend to update Java to 12 or 14 and set up ZGC.

 

[manikandan]

Hello Max,

Thank you for the reply. Memory and disk space is sufficient

I noticed from the client log that stream gets unpublished immediately after its gets to Publishing status

 

[Max]

Thank you for the reply. Memory and disk space is sufficient

How much publishers and subscribers you ususally handle on this server?

I noticed from the client log that stream gets unpublished immediately after its gets to Publishing status

This message shows that publishing was stopped by client, may be due to session disconnection. It seems like publisher channel issue.

 

[manikandan]

Thank you Max, I figured out the issue. The master origin server lost connectivity with the child origin servers which makes some logics to break in my application.

I mean when I ran command "cdn show-nodes" in master origin, child origin servers are not displayed there but it is active behind LB.

May I know what causes connection lost between those origin servers?

 

[Max]

May I know what causes connection lost between those origin servers?

Please check if tcp port 8084 is available in both directions (in and out) between master and child origins.
Also check if child origins displays each other using REST API query /cdn/show_nodes or using CLI.