CDN issue

[wyvasi]

Good day!
We have 2 origins and 2 edge servers.
Placeholder to describe the issue Origin: A,B Edge: C,D
Main server is A.
We redirect all traffic from B,D to A,C and we update the B and D -> we redirect traffic from A,C to B,D -> we update A and C ->
But now when we try to add back A and C, I can't find the nodes in the /rest-api/cdn/show_nodes so that I can reset outbound/inbound connection to all servers.
Is there anyway to fix this without restart B and D? (This would be the fix but I don't want to stop the streams)
Is there an endpoint or cli command to add new cdn nodes or force to connect to the server specified in cdn_point_of_entry?
Thanks!

 

[Max]

Good day.
Please clarify: do you have the only entry node per CDN (node A)?
In this case, when node A is restarted, you should either restart other nodes to enter CDN again, or try to reset all CDN connections on all the nodes using REST API: Connection reset using REST API

Is there an endpoint or cli command to add new cdn nodes or force to connect to the server specified in cdn_point_of_entry?

This is default behaviour: a CDN node tries to connect to cdn_point_of_entry node to enter CDN. If entry node is unreachable, a new nodes cannot enter to CDN.

 

[wyvasi]

Good day.
Please clarify: do you have the only entry node per CDN (node A)?

Yes only one cdn_point_of_entry: node A

Requests to node D and responses:

{
    "exception": "com.flashphoner.rest.server.exception.NotFoundException",
    "reason": "com.flashphoner.rest.server.exception.NotFoundException, Not Found, Node 10.1.1.140 not found, ts: 1648524907343, path: /rest-api/cdn/connection/reset_inbound",
    "path": "/rest-api/cdn/connection/reset_inbound",
    "error": "Not Found",
    "message": "Node A(tried both public and internal ip) not found",
    "timestamp": 1648524907343,
    "status": 404
}

{
    "exception": "com.flashphoner.rest.server.exception.NotFoundException",
    "reason": "com.flashphoner.rest.server.exception.NotFoundException, Not Found, Node 10.1.1.140 not found, ts: 1648525109219, path: /rest-api/cdn/connection/reset_outbound",
    "path": "/rest-api/cdn/connection/reset_outbound",
    "error": "Not Found",
    "message": "Node A(tried both public and internal ip) not found",
    "timestamp": 1648525109219,
    "status": 404
}

 

[Max]

Please check if TCP port 8084 is available both from D on A and from A on D nodes. This is CDN signaling port, it must be available on all nodes from all nodes.
Check also which nodes are connected to D using /cdn/show-nodes query.

 

[wyvasi]

Ports are fine, just tested again.
/cdn/show-nodes response from node D

[
    {
        "connectionStats": {
            "lastRtt": 1,
            "lastSn": 30107,
            "lastMessageSentMs": 1648527497600,
            "lastACKReceivedMs": 1648527497601,
            "maxRtt": 65536,
            "inFlightSize": 0
        },
        "version": "2.5",
        "role": "ORIGIN",
        "inboundConnected": false,
        "outboundConnected": true,
        "globalState": "ACTIVE",
        "processingState": "NEW_STREAMS_ALLOWED",
        "id": "ip of Node B"
    }
]

 

[Max]

Please provide SSH access to all the nodes using this form.

 

[wyvasi]

I am not allowed to do that. Will just restart everything.

 

[Max]

I am not allowed to do that. Will just restart everything.

Did restart helps?
Please also note that CDN consisting of Docker instances on the same host is not recommended for production/ We use that case for testing purposes only. If you use a such configuration consider migration to VPS or physical servers.

 

[wyvasi]

We use just one docker instance per host. But we need this fixed in future for 0 downtime deploy/update.

 

[Max]

We use just one docker instance per host. But we need this fixed in future for 0 downtime deploy/update.

To fix the problem, we need to reproduce it. That's why we asked to provide access to your environment.
We recommend the following:
1. Always update entry point node first.
2. If no connections to node in /cdn/show_nodes result:
2.1. Check if node is running
2.2. Check CDN port availability (8084/tcp by default) in both directions
2.3. Check server logs for connection failure messages
2.4. Check CDN configuration in flashphoner.properties file

 

[wyvasi]

To fix the problem, we need to reproduce it. That's why we asked to provide access to your environment.
We recommend the following:
1. Always update entry point node first.

I will try this next time, Thanks!
Everything is behind vpn and ssh keys.