Hello! there are any way to integrate WCS with a React native app ?

Max

Max

Administrator
Staff member
We didn't test with this platform.
As I know React Native compiles JavaScript / HTML code to native Java code for Android and native Objective C code for iOS. Let me know if I'm wrong.
However, core of Web SDK is WebRTC and it is browser based technology and I'm not sure it can be ported to React Native platform.
So, it seems you can port UI to React Native, but you can't port streaming engine and we have to provide a native plugin or extension to get this working on React Native.
 
J

Javier

Member
Max said:
We didn't test with this platform.
As I know React Native compiles JavaScript / HTML code to native Java code for Android and native Objective C code for iOS. Let me know if I'm wrong.
However, core of Web SDK is WebRTC and it is browser based technology and I'm not sure it can be ported to React Native platform.
So, it seems you can port UI to React Native, but you can't port streaming engine and we have to provide a native plugin or extension to get this working on React Native.
Click to expand...
Yes your are right, the way to have it working will be by giving a React Native library to implement FlashPhoner or WCS technology into React Native apps, I think it will be a great bet for Flashphoner because many people build apps using RN. BTW I had to use an own RTMP server and was able to get my expected goal. Cheers!
 
Max

Max

Administrator
Staff member
Hello.
We still have no news. Probably we'll move forward after new WebSDK major release.
 
Max

Max

Administrator
Staff member
There's no date yet due to more priority tasks (realtime mixer implementation, for example)
 
Jai Saravanan

Jai Saravanan

New Member
Max said:
Realtime mixer should allow to implement many-to-many conferences which is actual now. But this is not subject of this topic.
If React Native support is really needed, we recommend to fork WebSDK or use WCS raw Websocket API + WebRTC implementation
Click to expand...

Is there any working version for WCS raw Websocket API + WebRTC implementation?
Any github example or any workaround to implement the WCS raw Websocket API + WebRTC implementation.

Raw WebSocket API - Web Call Server 5.2 - EN - Flashphoner Documentation

docs.flashphoner.com docs.flashphoner.com
Where do we get the mediaSessionId, and SDP for the publishStream message(First step in the above URL)?
 
Last edited:
Max

Max

Administrator
Staff member
Is there any working version for WCS raw Websocket API + WebRTC implementation?
Click to expand...
Actually Web SDK is full implementation of Raw Websocket API + WebRTC

Any github example or any workaround to implement the WCS raw Websocket API + WebRTC implementation.
Click to expand...
github.com

GitHub - flashphoner/flashphoner_client: Client for Flashphoner-server

Client for Flashphoner-server. Contribute to flashphoner/flashphoner_client development by creating an account on GitHub.
github.com github.com

Where do we get the mediaSessionId
Click to expand...
MediaSessionId is UUID generated on the JavaScript end
github.com

flashphoner_client/src/flashphoner-core.js at wcs_api-2.0 · flashphoner/flashphoner_client

Client for Flashphoner-server. Contribute to flashphoner/flashphoner_client development by creating an account on GitHub.
github.com github.com
Code: 
 var id_ = uuid_v1();
Here you can see how websocket message filled and send
github.com

flashphoner_client/src/flashphoner-core.js at wcs_api-2.0 · flashphoner/flashphoner_client

Client for Flashphoner-server. Contribute to flashphoner/flashphoner_client development by creating an account on GitHub.
github.com github.com
Code: 
 }).then(function (offer) {
                    logger.debug(LOG_PREFIX, "Offer SDP:\n" + offer.sdp);
                    //publish stream with offer sdp to server
                    send("publishStream", {
                        mediaSessionId: id_,
                        name: name_,
                        published: published_,
and SDP for the publishStream message
Click to expand...
To get SDP you have to pass constraints to peerConnection.createOffer();
github.com

flashphoner_client/src/webrtc-media-provider.js at 73a783f61e30049e9c863f238025555ae8cdf835 · flashphoner/flashphoner_client

Client for Flashphoner-server. Contribute to flashphoner/flashphoner_client development by creating an account on GitHub.
github.com github.com
Once promise is resolved you have to get offer.sdp from browser WebRTC API.
Code: 
connection.createOffer(constraints).then(function (offer) {
                    //WCS-2919 Workaround for Chromium bug to play stereo
                    if (options.stereo) {
                        offer.sdp = offer.sdp.replace('minptime=10', 'minptime=10;stereo=1;sprop-stereo=1');
                    }
                    connection.setLocalDescription(offer).then(function () {
                        var o = {};
                        o.sdp = util.stripCodecs(offer.sdp, options.stripCodecs);
                        o.hasAudio = hasAudio;
                        o.hasVideo = hasVideo;
                        resolve(o);
                    });
                });
 
Jai Saravanan

Jai Saravanan

New Member
As per documentation, "Playstream" message has been sent to WCS and I got the response of "notifyAudioCodec", "setRemoteSDP", and "notifyStreamStatusEvent".

But I didn't get the response of "notifyVideoFormat" and the stream is not playing is just an empty page. It didn't trigger the "onaddstream" callback function of RTCPeerConnection.

Could you guide me if anything I am missing?

1663843276178.png

I have added the code:

JavaScript: 
const App = () => {
  const [stream, setStream] = React.useState<MediaStream>();
  const [remoteStream, setRemoteStream] = React.useState<MediaStream>();
  const [wsConnection, setWsConnection] = React.useState(
    new WebSocket('wss://demo.flashphoner.com:8443/'),
  );
  const [rtcConnection, setRtcConnection] = React.useState<RTCPeerConnection>(
    new RTCPeerConnection({
      iceServers: [],
    }),
  );
  const [initialConnectCompleted, setInitialConnectCompleted] =
    React.useState<boolean>(false);
  let isFront = true;
  const start = async () => {
    console.log(' *** start *** ');
    if (!stream) {
      let s: any;
      try {
        s = await mediaDevices.getUserMedia({
          audio: true,
          video: {facingMode: isFront ? 'user' : 'environment'},
        });
        setStream(s);
      } catch (e) {
        if (e instanceof Error) {
          console.log(e.message);
        }
      }
    }
  };
  const connectingWebsocket = () => {
    console.log(' *** Initial connect message *** ');
    // sending initial connection message.
    sendMessage({
      data: [
        {
          appKey: 'defaultApp',
          clientBrowserVersion: null,
          clientOSVersion: UserAgent.getUserAgent(),
          clientVersion: UserAgent.applicationVersion,
          mediaProviders: ['WebRTC', 'MSE', 'WSPlayer'],
          msePacketizationVersion: 2,
        },
      ],
      message: 'connection',
    });
  };
  const sendMessage = (message: any) => {
    console.log(' *** message *** ', message);
    wsConnection.send(JSON.stringify(message));
  };
  const stop = () => {
    console.log(' *** stop *** ');
    if (stream) {
      stream.release();
      setStream(undefined);
    }
  };
  const createRtcOffer = () => {
    console.log(' *** createRtcOffer *** ');
    setInitialConnectCompleted(true);
    var constraints: RTCOfferOptions = {
      offerToReceiveAudio: true,
      offerToReceiveVideo: true,
    };
    rtcConnection.createOffer(constraints).then(function (offer) {
      console.log(' *** Offer *** ', offer);
      console.log(' *** setLocalDescription *** ');
      rtcConnection.setLocalDescription(offer).then(function () {
        var o: any = {};
        o.sdp = offer.sdp;
        o.hasAudio = true;
        o.hasVideo = true;
        sendOffer(offer);
      });
    });
  };
  const sendOffer = async (offer: any) => {
    console.log(' *** sendOffer *** ');
    try {
      if (rtcConnection) {
        sendMessage({
          data: [
            {
              bitrate: 0,
              hasAudio: true,
              hasVideo: true,
              height: 0,
              maxBitrate: 0,
              mediaProvider: 'WebRTC',
              mediaSessionId: uuid_v1(),
              minBitrate: 0,
              name: '9e18',
              published: false,
              quality: 0,
              record: false,
              sdp: offer.sdp,
              status: 'PENDING',
              width: 0,
            },
          ],
          message: 'playStream',
        });
      }
    } catch (err) {
      if (err instanceof Error) {
        console.log('Offerr Error: ', err.message);
      }
    }
  };
  const sendPongMessage = () => {
    console.log(' *** sendPongMessage *** ');
    sendMessage({
      message: 'pong',
      data: [null],
    });
  };
  React.useEffect(() => {
    start();
    //websocketConnection callback
    wsConnection.onopen = () => {
      console.log('*** Connected to the signaling server *** ');
      connectingWebsocket();
    };
    wsConnection.onerror = function (err) {
      if (err instanceof Error) {
        console.log('!!!Got error on webSocket!!! : ', err.message);
      }
    };
    wsConnection.onmessage = (event: any) => {
      const messageData = JSON.parse(event.data);
      console.log(' *** onMessage: *** ', messageData);
      if (messageData.message === 'ping') {
        sendPongMessage();
      }
      if (messageData.message === 'getVersion') {
        console.log('is it coming here!');
        if (!initialConnectCompleted) {
          createRtcOffer();
        }
      }
    };
    //rtcConnection callback
    rtcConnection.onaddstream = (event: any) => {
      console.log(' *** on add stream *** ');
      setRemoteStream(event.stream);
    };
    rtcConnection.onremovestream = () => console.log('stream removed');
    rtcConnection.onconnectionstatechange = (event: any) => {
      console.log(
        'state change connection >>>>: ',
        rtcConnection?.connectionState,
      );
    };
    rtcConnection.onsignalingstatechange = () =>
      console.log('onsignalingstatechange: ', rtcConnection?.signalingState);
    rtcConnection.onicecandidateerror = error => console.log(error);
    return () => {
      stop();
    };
  }, [wsConnection, initialConnectCompleted, rtcConnection]);
  return (
    <>
      {remoteStream && (
        <RTCView streamURL={remoteStream.toURL()} style={styles.stream} />
      )}
    </>
  );
};

Debugger Log:
1663841632779.png
 
Last edited:
Jai Saravanan

Jai Saravanan

New Member
Similarly in "publishStream", getting failed state in the "onconnectionstatechange" callback function of RTCPeerConnection. Added the screenshot below.

1663855997205.png


Added the stream code below:

JavaScript: 
const App = () => {
  const uuid = uuid_v1();
  const [stream, setStream] = React.useState<MediaStream>();
  const [remoteStream, setRemoteStream] = React.useState<MediaStream>();
  const [wsConnection, setWsConnection] = React.useState(
    new WebSocket('wss://demo.flashphoner.com:8443/'),
  );
  const [rtcConnection, setRtcConnection] = React.useState<RTCPeerConnection>(
    new RTCPeerConnection({
      iceServers: [],
    }),
  );
  const [initialConnectCompleted, setInitialConnectCompleted] =
    React.useState<boolean>(false);
  let isFront = true;

  const start = async () => {
    console.log(' *** start *** ');
    if (!stream) {
      let s: any;
      try {
        s = await mediaDevices.getUserMedia({
          audio: true,
          video: {facingMode: isFront ? 'user' : 'environment'},
        });
        setStream(s);
      } catch (e) {
        if (e instanceof Error) {
          console.log(e.message);
        }
      }
    }
  };

  const connectingWebsocket = () => {
    console.log(' *** Initial connect message *** ');
    // sending initial connection message.
    sendMessage({
      data: [
        {
          appKey: 'defaultApp',
          clientBrowserVersion: null,
          clientOSVersion: UserAgent.getUserAgent(),
          clientVersion: UserAgent.applicationVersion,
          mediaProviders: ['WebRTC', 'MSE', 'WSPlayer'],
          msePacketizationVersion: 2,
        },
      ],
      message: 'connection',
    });
  };

  const sendMessage = (message: any) => {
    console.log(' *** message *** ', message);
    wsConnection.send(JSON.stringify(message));
  };

  const stop = () => {
    console.log(' *** stop *** ');
    if (stream) {
      stream.release();
      setStream(undefined);
    }
  };

  const createRtcOffer = () => {
    console.log(' *** createRtcOffer *** ');
    setInitialConnectCompleted(true);
    var constraints: RTCOfferOptions = {
      offerToReceiveAudio: true,
      offerToReceiveVideo: true,
    };
    rtcConnection.createOffer(constraints).then(function (offer) {
      //WCS-2919 Workaround for Chromium bug to play stereo
      // offer.sdp = offer.sdp.replace(
      //   'minptime=10',
      //   'minptime=10;stereo=1;sprop-stereo=1',
      // );
      console.log(' *** Offer *** ', offer);
      console.log(' *** setLocalDescription *** ');
      rtcConnection.setLocalDescription(offer).then(function () {
        var o: any = {};
        o.sdp = stripCodecs(offer.sdp, '');
        o.hasAudio = true;
        o.hasVideo = true;
        sendOffer(offer);
      });
    });
  };

  const setRemoteDesc = (sdp: string) => {
    console.log(' *** setRemoteDescription *** ');
    const remoteAnswer = {
      type: 'answer',
      sdp: sdp,
    };
    rtcConnection.setRemoteDescription(remoteAnswer).then(function () {
      console.log('remote description set!');
    });
  };

  const sendOffer = async (offer: any) => {
    console.log(' *** sendOffer *** ');
    try {
      if (rtcConnection) {
        sendMessage({
          data: [
            {
              bitrate: 0,
              hasAudio: true,
              hasVideo: true,
              maxBitrate: 0,
              mediaProvider: 'WebRTC',
              mediaSessionId: uuid,
              minBitrate: 0,
              name: 'ja14',
              published: true,
              record: false,
              sdp: offer.sdp,
              status: 'PENDING',
            },
          ],
          message: 'publishStream',
        });
      }
    } catch (err) {
      if (err instanceof Error) {
        console.log('Offerr Error: ', err.message);
      }
    }
  };

  const sendPongMessage = () => {
    console.log(' *** sendPongMessage *** ');
    sendMessage({
      message: 'pong',
      data: [null],
    });
  };

  React.useEffect(() => {
    start();
    //websocketConnection callback
    wsConnection.onopen = () => {
      console.log('*** Connected to the signaling server *** ');
      connectingWebsocket();
    };
    wsConnection.onerror = function (err) {
      if (err instanceof Error) {
        console.log('!!!Got error on webSocket!!! : ', err.message);
      }
    };
    wsConnection.onmessage = (event: any) => {
      const messageData = JSON.parse(event.data);
      console.log(' *** onMessage: *** ', messageData);
      if (messageData.message === 'ping') {
        sendPongMessage();
      }
      if (messageData.message === 'getVersion') {
        console.log('is it coming here!');
        if (!initialConnectCompleted) {
          createRtcOffer();
        }
      }
      if (messageData.message === 'setRemoteSDP') {
        if (messageData.data.length && messageData.data[1]) {
          setRemoteDesc(messageData.data[1]);
        }
      }
    };

    //rtcConnection callback
    rtcConnection.onaddstream = (event: any) => {
      console.log(' *** on add stream *** ');
      setRemoteStream(event.stream);
    };
    rtcConnection.onremovestream = () => console.log('stream removed');
    rtcConnection.onconnectionstatechange = (event: any) => {
      console.log(
        'onconnectionstatechange >>>>: ',
        rtcConnection?.connectionState,
      );
    };
    rtcConnection.onsignalingstatechange = () =>
      console.log('onsignalingstatechange: ', rtcConnection?.signalingState);
    rtcConnection.onicecandidateerror = error => console.log(error);

    return () => {
      stop();
    };
  }, [wsConnection, initialConnectCompleted, rtcConnection]);

  const stripCodecs = function (sdp: any, codecs: any) {
    if (!codecs.length) return sdp;
    var sdpArray = sdp.split('\n');
    var codecsArray = codecs.split(',');
    //search and delete codecs line
    var pt = [];
    var i;
    for (var p = 0; p < codecsArray.length; p++) {
      console.log('Searching for codec ' + codecsArray[p]);
      for (i = 0; i < sdpArray.length; i++) {
        if (
          sdpArray[i].search(new RegExp(codecsArray[p], 'i')) != -1 &&
          sdpArray[i].indexOf('a=rtpmap') == 0
        ) {
          console.log(codecsArray[p] + ' detected');
          pt.push(sdpArray[i].match(/[0-9]+/)[0]);
          sdpArray[i] = '';
        }
      }
    }
    if (pt.length) {
      //searching for fmtp
      for (p = 0; p < pt.length; p++) {
        for (i = 0; i < sdpArray.length; i++) {
          if (
            sdpArray[i].search('a=fmtp:' + pt[p]) != -1 ||
            sdpArray[i].search('a=rtcp-fb:' + pt[p]) != -1
          ) {
            sdpArray[i] = '';
          }
        }
      }
      //delete entries from m= line
      for (i = 0; i < sdpArray.length; i++) {
        if (
          sdpArray[i].search('m=audio') != -1 ||
          sdpArray[i].search('m=video') != -1
        ) {
          var mLineSplitted = sdpArray[i].split(' ');
          var newMLine = '';
          for (var m = 0; m < mLineSplitted.length; m++) {
            if (pt.indexOf(mLineSplitted[m].trim()) == -1 || m <= 2) {
              newMLine += mLineSplitted[m];
              if (m < mLineSplitted.length - 1) {
                newMLine = newMLine + ' ';
              }
            }
          }
          sdpArray[i] = newMLine;
        }
      }
    }
    //normalize sdp after modifications
    var result = '';
    for (i = 0; i < sdpArray.length; i++) {
      if (sdpArray[i] != '') {
        result += sdpArray[i] + '\n';
      }
    }
    return result;
  };

  return (
    <>
      {stream && <RTCView streamURL={stream.toURL()} style={styles.stream} />}
    </>
  );
};
 
Max

Max

Administrator
Staff member
When publishing a stream, publishStream should be sent, then setRemoteSDP and notifyStreamStatusEvent are received.
1663895646372.png

Seems like you're receiving the messages above, so signaling is working. But, if onconnectionstatechange returns failed state, it means WebRTC connection is not established. This may be due to media ports availability issues. Please check if outgoing connections to UDP ports 31001-32000 are not locked on the device you're testing on. See also this doc about WebRTC traffic analyzing.
You can also check notifyStreamStatusEvent payload for error status.
When playing a stream, publishStream should be sent, and the following messages should be received:
1663895805606.png

If you didn't receive something after notifyStreamStatusEvent, please check its payload. You have probably
Code: 
...
info: "Session does not exist",
status: "FAILED",
...
or some other error message. The example above means the stream with such name is not published on server.
 
Jai Saravanan

Jai Saravanan

New Member
Exact error message while I am trying to publish the stream.

Already this issue has been reported in another forum.

Failed by DTLS error

Hello. I am getting failed by DTLS error intermittently. The code works well from desktop browsers (Chrome and Firefox) bur fails in Android Chrome and Firefox. I am using TLS version 1.2 on my server. Please help me solve this issue. It's impacting our business big time.
forum.flashphoner.com

1663918301105.png



As you say, do I need to check all our app users' devices' "outgoing connections to UDP ports 31001-32000" whether it is open or not?
Seems like you're receiving the messages above, so signaling is working. But, if onconnectionstatechange returns failed state, it means WebRTC connection is not established. This may be due to media ports availability issues. Please check if outgoing connections to UDP ports 31001-32000 are not locked on the device you're testing on. See also this doc about WebRTC traffic analyzing.
Click to expand...
 
You must log in or register to reply here.
Top