Our server is being spammed

[Azhar Ali]

Hiya,

We are trying to deal with attack of some sort but don't whats wrong and how to deal with. We have restarted the server several times but its no use.
here is a stats
Can you please help to identify ip or something so we can block them. Currently, its unusable
Already emailed the ssh access
Please help as soon as its possible.
-----Connection Stats-----
connections=9879
connections_rtmfp=0
connections_websocket=9879
-----Port Stats-----
ports_media_free=9998
ports_media_busy=0
ports_media_quarantine=0
-----Stream Stats-----
streams_webrtc_in=0
streams_webrtc_out=0
streams_rtmfp_in=0
streams_rtmfp_out=0
streams_rtmp_in=0
streams_rtmp_out=0
streams_viewers=
streams_rtsp_in=0
streams_rtsp_out=0
streams_websocket_out=0
streams_rtmp_client_out=0
streams_play_rate=0
streams_stop_rate=0
-----Native Resources-----
native_resources=
-----Core Stats-----
core_java_threads=190
core_java_committedMemory=19810496512
core_java_freePhysicalMemorySize=65201999872
core_java_arch=amd64
core_java_availableProcessors=36
core_java_freeSwapSpaceSize=0
core_java_maxFileDescriptorCount=20000
core_java_open_file_descriptors=19997
core_java_cpu_usage=1.08
core_java_totalPhysicalMemorySize=73591664640
core_java_totalSwapSpaceSize=0
core_java_uptime=215710
-----Call Stats-----
sip_processed_calls=0
sip_calls_state=established/0,trying/0,ringing/0,ring/0,ring_media/0,hold/0,busy/0,finish/0,session_progress/0,pending/0,failed/0
sip_calls=0
sip_calls_established=0
sip_calls_in=0
sip_calls_out=0
sip_calls_per_second=0.00
-----Sip Stats-----
sip_registered=0
-----Recording Stats-----
recording_sessions=0
-----System Stats-----
system_java_cpu_usage=1.40
system_java_load_average=0.14
-----Network Stats (Mbit/s)-----
global_bandwidth_in=0.000
global_bandwidth_out=0.000
-----Version info-----
wcs_version=5.2.213-84010b55b1c22b65ef22db69ff5c119bf8a65ece
wcs_client_version=0.5.28.2753-da42595a3e045b0bf292eda0af43dff1f17dfd83
-----Errors info-----
java.net.SocketException=261
java.lang.NoSuchMethodException=45
org.jboss.netty.channel.ChannelException=8
java.io.IOException=224
java.net.UnknownHostException=42
java.io.FileNotFoundException=124
-----Degraded streams-----
degraded_streams=
degraded_streams_percent=0

 

[Max]

Hello

connections_websocket=9879

As you can see you have a lot of connections to websocket port (8443 by default)
So you have to protect TCP connections to this port using
- netstat
- iptables
- tcpdump
Using netstat you can find established connections

netstat -nlp | grep java

Using iptables you can reject unwanted IPs

iptables -A INPUT -s 172.16.1.100 -j DROP

Using tcpdump you can trace active traffic

tcpdump port 8443

 

[Azhar Ali]

thanks Max.
we are also building some authorisation using backed rest hook. I have manged to get most of working for the defaultApp for the normal streams.
But, we also have room and using roomAPI and I think for that the app is RoomApp, once I point that app to our backend domain for authentication it stops working.
I have implemented the following rest methods:
/connect
/publishStream
/stopStream
/StreamStatusEvent
/ConnectionStatusEvent
/OnDataEvent
Apart from the connect method I am returning nothing with http 200 Ok

Any ideas?

 

[Max]

Good day.
Unfortunately, we cannot connect to your server with the credentials that you've sent, so we can't check your setup.
Please clarify if room connection works when your backend return 200 OK in /connect method. Also, please collect a report as described here for the following two cases:
1) backend returns 200 OK on /connect query when user tries to join the room
2) backend returns 403 Forbidden on /connect query when user tries to join the room
and send us, we will check.

 

[Azhar Ali]

Hello Max,

Thank you for getting back to us.
Yes we have changed the domain for the server due to the above problem. Essentially you can replace xxxxx3.xxx.com to 5 and rest is the same.
I had to change the rest domain back to the localhost:8081/apps/RoomApp because we need to have this available on week days between 9am to 4pm ET.
To answer your question:
1) Yes room connect was working and calling the rest /connect and we send the same response back as incoming data. (this is same connect hook which works fine for normal streams). We currently arn't doing any authorisation checks in the hook as its in test mode, we are just trying to get the flow working.
At start, I only implemented the /connect and looked at the server_logs and there was 404 errors on ConnectionStatusEvent and OnDataEvent as I didn't implemented them first. Once I did them, the 404 error went away but room still wouldn't work.

Flashphoner.roomApi.connect

works and callback to SESSION_STATUS.ESTABLISHED is also called but after calling connection.join({ name: roomName }) none of the callbacks would fire, no join left or failed. Its like it gets hanged there.
Do you think its possible to use our hook url to another demo system to check if there is anything wrong with it? I can email the url to support?

 

[Max]

RoomApi is a special wrapper for WCS signaling to implement conference room. It is build over REST hook using default backend to send messages to all participants, publish video and so on. So if you change default application to your own, RoomApi functions stops working. You can either use RoomApi or implement REST hook authentication but not both.
However, you have three options here:
1) Implement your own chat rooms using REST hooks.
2) Implement user authetication on your own backend routing all the queries to default roomApp application http://localhost:8081/apps/RoomApp and then routing all the responses back to WCS. Your backend shoud be a transparent proxy in this case.
3) Implement video conference using MCU support in stream mixer. This requires more server CPU, but it can be much simpler to implement.

 

[Azhar Ali]

We have used the roomAPI in order to broadcast a radio. There is no video or text chat. It might not have been the best option based on what we required but this is how its done at the moment.
We required only one broadcaster to stream its voice to all the connected users. Users would be connected to the room and using join / leave event we can start the audio stream.
What we have is working nicely in our site except there is no security on it. Stream mixer seems overkill for this use case.
Would you able to guide us through option 2?

 

[Max]

Good day.
In your case, even RoomApi seems excessive. If you have many listeners per one broadcaster with no any chat between them, it is basic WebRTC streaming case. You can just publish WebRTC audio stream as TwoWayStreaming example does (with constraints {video: false, audio: true} of course), then play it as Player example does (again, using constraints for audio only). REST hooks work in this case, and all users authentication is yours.
But, if you choose more complex way with RoomApi, you can tray to implement REST hooks passing all queries to WCS default RoomApp as follows:

<?php
$api_method = array_pop(explode("/", $_SERVER['REQUEST_URI']));
$incoming_data = json_decode(file_get_contents('php://input'), true);
$response_data = $incoming_data;

switch($api_method) {
    case"connect":
        // Authenticate user as you wish
        if (authenticateUser($incoming_data)) {
            error_log("User authorized");
            // POST /connect query to WCS default RoomApp
            $response_data = postToRoomApp(json_encode($incoming_data));
        } else {
            error_log("User not authorized. Connection failed with 403 status.");
            ubnormalResponse(403);
        }
    break;
    default:
        // POST any query to WCS default RoomApp
        $response_data = postToRoomApp(json_encode($incoming_data));
    break;
}
header('Content-Type: application/json');
echo json_encode($response_data);

// User authentication
function authenticateUser($incoming_data) {
...
}

// Passing query to WCS default RoomApp
function postToRoomApp($data_string) {
    // Here should be your WSC RoomApp address
    $wcs_url="http://wcs:8081/apps/RoomApp";

    $response="";
    if( $curl = curl_init() ) {
        curl_setopt($curl, CURLOPT_URL, $wcs_url);
        curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
        curl_setopt($curl, CURLOPT_POST, true);
        curl_setopt($curl, CURLOPT_CUSTOMREQUEST, "POST");
        curl_setopt($curl, CURLOPT_POSTFIELDS, $data_string);
        curl_setopt($curl, CURLOPT_HEADER, true);
        curl_setopt($curl, CURLOPT_HTTPHEADER, array('Content-Type: application/json',                                                                               
                               'Content-Length: ' . strlen($data_string)));      
        $response = curl_exec($curl);
        $http_code = curl_getinfo($curl, CURLINFO_HTTP_CODE);
        if($http_code != 200) {
            ubnormalResponse($http_code);
        }
        curl_close($curl);
    }
    return($response);
}
function ubnormalResponse($code) {
    if ($code == 403) {
    header('HTTP/1.1 403 Forbidden', true, $code);
    } else {
    header(':', true, $code);
    }
    die();
}
?>