Cannot play mixer stream in iOS

[kindco]

Hello,
We cannot get mixer video playing in iOS. After stream is created, we get a black video area. Then after a minute "Failed By ICE timeout" error appears. We tested this on various iOS devices, so it seems to be the general error. Note that "Two-way Streaming" example works, so this is either an error in our code or something mixer-related.

These are 2 methods in our live meeting React class that start and play mixer stream:

/**
 * Creates and plays mixer stream.
 */
playMixerStream () {
  // This makes everything work in iOS Safari.
  if (IS_SAFARI && navigator.mediaDevices || Flashphoner.getMediaProviders()[0] === 'MSE') {
    Flashphoner
      .playFirstVideo(this.mixerRef.current, false, this.preloaderVideo)
      .then(() => {
        // Create a mixer stream.
        this.createMixerStream()
        // Play created stream.
        this.mixerStream.play()
      })
      .catch((e) => {
        this.setError(e.message)
        console.error(e)
      })
  }
  else {
    // Create a mixer stream.
    this.createMixerStream()
    // Play created mixer stream.
    this.mixerStream.play()
  }
}


/**
 * Creates WCS mixer stream.
 */
createMixerStream () {
  const { mixerName, streamName } = this.props

  // Create a mixer stream.
  // noinspection JSUnresolvedVariable
  this.mixerStream = this.wcsSession.createStream({
    name: mixerName + '-' + streamName,
    display: this.mixerRef.current
  })

  // Stream is created and is playing.
  // noinspection JSUnresolvedVariable
  this.mixerStream.on(STREAM_STATUS.PLAYING, () => {
    this.setStatus('live')
  })

  // Stream is failed for some reason.
  this.mixerStream.on(STREAM_STATUS.FAILED, (stream) => {
    const error = this.getStreamError(stream)
    if (error === 'Stopped by publisher stop') {
      this.setStatus(MIXER_POST_ACTION_STATUSES['stop'])
    }
    else {
      this.setError(this.getStreamError(stream))
    }
  })
}

Note that we checked condition if (IS_SAFARI...) and it worked well.

I cannot see any essential difference between our code and Two-way streaming example. Maybe you can? Thank you!

 

[Max]

Good day.
Please clarify: can the same mixer output stream on the same device be played in Two Way Streaming example (Player section) or in Player example?
If not, this seems to be media ports blocking issue, according to Failed by ICE timeout error info.
Please check the following:
- if media ports (31001-32000/udp by default) are available on server from the device/network used
- if the stream you're trying to play is available on server
Read here how to check ports availability.
Use /stream/find REST API query to check if stream is published on server:

POST /rest-api/stream/find HTTP/1.1
Host: wcs:8081
Content-Type: application/json
 
{
    "name":"mixer-stream",
    "published":true
}

 

[kindco]

Hello @Max ,
The Two-way Streaming example works for mixer streams on iPhone, so it's something with our code.
Do you have any thoughts what can be wrong with it?

 

[Max]

Do you have any thoughts what can be wrong with it?

Please add debug logging to check the flow. Seems like either playFirstVideo and stream playback work in different contexts or playFirstVideo is not invoked at all.

 

[kindco]

Hello @Max

Please add debug logging to check the flow.

What logging do you mean? Server logs? Some kind of front-end logging?

 

[Max]

What logging do you mean? Server logs? Some kind of front-end logging?

Frontend logging to the browser console like:

  if (IS_SAFARI && navigator.mediaDevices || Flashphoner.getMediaProviders()[0] === 'MSE') {
    Flashphoner
      .playFirstVideo(this.mixerRef.current, false, this.preloaderVideo)
      .then(() => {
        console.log("playFirstVideo resolved");
        // Create a mixer stream.
        this.createMixerStream()
        // Play created stream.
        this.mixerStream.play()
      })
      .catch((e) => {
        this.setError(e.message)
        console.error(e)
      })
  }
...
createMixerStream () {
  ...
  // Create a mixer stream.
  // noinspection JSUnresolvedVariable
  console.log("create a stream");
  this.mixerStream = this.wcsSession.createStream({
    name: mixerName + '-' + streamName,
    display: this.mixerRef.current
  })
...
}

 

[kindco]

Hello @Max

We added console.log() calls almost everywhere:

/**
 * Creates and plays mixer stream.
 */
playMixerStream () {
  console.log('Entering playMixerStream()')
  console.log('Is Safari: ' + this.isSafari)
  if (Flashphoner.getMediaProviders()[0] === 'WSPlayer') {
    console.log('WSPlayer detected')
    Flashphoner.playFirstSound()
  }
  // This makes everything work in iOS Safari.
  else if (this.isSafari || Flashphoner.getMediaProviders()[0] === 'MSE') {
    console.log('Safari/MSE detected')
    console.log('Playing ' + this.preloaderVideo)
    Flashphoner
      .playFirstVideo(this.mixerRef.current, false, this.preloaderVideo)
      .then(() => {
        console.log('playFirstVideo() resolved')
        // Create a mixer stream.
        this.createMixerStream()
        console.log('About to play')
        // Play created stream.
        this.mixerStream.play()
      })
      .catch((e) => {
        this.setError(e.message)
        console.error(e)
      })
    return
  }

  // Create a mixer stream.
  this.createMixerStream()
  // Play created mixer stream.
  this.mixerStream.play()
}

/**
 * Creates WCS mixer stream.
 */
createMixerStream () {
  const { mixerName, streamName } = this.props
  console.log('About to create a stream')

  // Create a mixer stream.
  // noinspection JSUnresolvedVariable
  this.mixerStream = this.wcsSession.createStream({
    name: mixerName + '-' + streamName,
    display: this.mixerRef.current
  })

  console.log('Stream created')

  // Stream is created and is playing.
  // noinspection JSUnresolvedVariable
  this.mixerStream.on(STREAM_STATUS.PLAYING, () => {
    console.log('Stream playing')
    this.setStatus('live')
  })

  // Stream is failed for some reason.
  this.mixerStream.on(STREAM_STATUS.FAILED, (stream) => {
    console.log('Stream failed')
    const error = this.getStreamError(stream)
    if (error === 'Stopped by publisher stop') {
      this.setStatus(MIXER_POST_ACTION_STATUSES['stop'])
    }
    else {
      this.setError(this.getStreamError(stream))
    }
  })
}

And here is the log we got:


As you can see, playFirstVideo() works, but the stream still fails at the end.

 

[Max]

As you can see, playFirstVideo() works, but the stream still fails at the end.

What Stream.getInfo() and [ICODEStream.getErrorInfo()[/ICODE] methods return in this case?

 

[kindco]

Hello @Max

We added logs for getInfo() and getErrorInfo():

this.mixerStream.on(STREAM_STATUS.FAILED, (stream) => {
  console.log('Stream failed')
  console.log('getInfo:', stream.getInfo())
  console.log('getErrorInfo:', stream.getErrorInfo())
  const error = this.getStreamError(stream)
  if (error === 'Stopped by publisher stop') {
    this.setStatus(MIXER_POST_ACTION_STATUSES['stop'])
  }
  else {
    this.setError(this.getStreamError(stream))
  }
})

Unfortunately, not too much info here:

 

[Max]

Failed by ICE timeout means WebRTC connection cannot be established from the device to the server. It is no depending on client code. So we back to the post above:
- please check if media ports (31001-32000/udp by default) are avalable from the device where error occurs
- please collect a report on server as described here including client debug logs and send using this private form.

 

[kindco]

Hello @Max
I'm trying to generate a report with sudo ./report.sh --sysinfo --conf --tar, but getting an error Error: Process 2607 does not exist. What can be the issue?

 

[Max]

I'm trying to generate a report with sudo ./report.sh --sysinfo --conf --tar, but getting an error Error: Process 2607 does not exist. What can be the issue?

Seems like Java process is stopped. Please check if WCS is running before collecting a report archive. If WCS is running, and issue still persists, please provide SSH access to the server using this private form, we will check.

 

[Max]

We've checked you server and found lock file /var/run/FlashphonerWebCallServer.pid from previous installation. Without it, report.sh works correctly (claiming to lsof absence, but this is not mandatory).
But logs are not writing. So we recommend to do the following:
1. Stop WCS.

systemctl stop webcallserver

2. Clean logs folder

rm -rf /usr/local/FlashphonerWebCallServer/logs/*

3. Start WCS

systemctl stop webcallserver

Then, reproduce the "Failed by ICE timeout" issue, collect a report

sudo ./report.sh --sysinfo --conf --tar

and send using this private form.

 

[kindco]

OK, done. Thank you very much!

 

[Max]

Please recollect the logs with INFO log level in log4j.properties file

log4j.rootLogger=info, stdout, fAppender

Not enough information with WARN level.

 

[kindco]

Sorry, we were preparing to production and changed this setting. Will recollect the logs. Thank you!

 

[kindco]

Hello,

I tried running ./report.sh --sysinfo --conf --tar from /usr/local/FlashphonerWebCallServer/bin but that script doesn't exist in that location. I wasn't able to find it using "find" either, what is the correct location of this script?

 

[Max]

report.sh is in /usr/local/FlashphonerWebCallServer/tools folder:

cd /usr/local/FlashphonerWebCallServer/tools
sudo ./report.sh --sysinfo --conf --tar

Please see the doc.

 

[kindco]

@Max
Sorry for confusion. Report is prepared and sent.

 

[Max]

We checked the report.
1. About server config
Your server config is too weak for mixing. We now working on mixer load testing and plan to release an article to our blog, please stay tuned. Preliminary results is 2 CPU core per 1 mixer. You're trying to run mixer with default output resolution and bitrate on server instance with 1 CPU core. In addition to mixing, you're republishing output mixer stream to RTMP server which leads to audio transcoding from Opus to AAC. So CPU is probably overloaded.
Also, you have only 2 Gb RAM (1 Gb per Java heap). When mixer is running, garbage collector works every 2 seconds which is too often because every time Java machine stops.
So we recommend you to test on server with at least 2 CPU cores (if only one mixer is used in tests) and 8 Gb RAM (4 Gb per Java heap). The closest config of basic DO droplets is 4 CPU/8 Gb RAM.
2. About server setup
You need to enable multithreaded mixer setting only if you have 10 participants in mixer and more. So you should disable it espacially on weak server.
Also, you're trying to raise publishing bitrate dramatically

webrtc_cc_min_bitrate = 1500000
webrtc_cc_max_bitrate = 10000000
webrtc_sdp_min_bitrate_bps = 1500000
webrtc_sdp_max_bitrate_bps = 7000000

This may be useful only for 4K streaming through the good channels (20 Mbps). In any other case this may lead to huge packet losses and publishing quality drop (freezes, artifacts, video traffic stopping). We recommend to set lower bitrate boundaries to 100000 bps or leave them as default.
So we recommend to solve the above problems, and then test your code again.