Cannot play mixer stream in iOS

kindco

Member
Hello,
We cannot get mixer video playing in iOS. After stream is created, we get a black video area. Then after a minute "Failed By ICE timeout" error appears. We tested this on various iOS devices, so it seems to be the general error. Note that "Two-way Streaming" example works, so this is either an error in our code or something mixer-related.

These are 2 methods in our live meeting React class that start and play mixer stream:

JavaScript:
/**
 * Creates and plays mixer stream.
 */
playMixerStream () {
  // This makes everything work in iOS Safari.
  if (IS_SAFARI && navigator.mediaDevices || Flashphoner.getMediaProviders()[0] === 'MSE') {
    Flashphoner
      .playFirstVideo(this.mixerRef.current, false, this.preloaderVideo)
      .then(() => {
        // Create a mixer stream.
        this.createMixerStream()
        // Play created stream.
        this.mixerStream.play()
      })
      .catch((e) => {
        this.setError(e.message)
        console.error(e)
      })
  }
  else {
    // Create a mixer stream.
    this.createMixerStream()
    // Play created mixer stream.
    this.mixerStream.play()
  }
}


/**
 * Creates WCS mixer stream.
 */
createMixerStream () {
  const { mixerName, streamName } = this.props

  // Create a mixer stream.
  // noinspection JSUnresolvedVariable
  this.mixerStream = this.wcsSession.createStream({
    name: mixerName + '-' + streamName,
    display: this.mixerRef.current
  })

  // Stream is created and is playing.
  // noinspection JSUnresolvedVariable
  this.mixerStream.on(STREAM_STATUS.PLAYING, () => {
    this.setStatus('live')
  })

  // Stream is failed for some reason.
  this.mixerStream.on(STREAM_STATUS.FAILED, (stream) => {
    const error = this.getStreamError(stream)
    if (error === 'Stopped by publisher stop') {
      this.setStatus(MIXER_POST_ACTION_STATUSES['stop'])
    }
    else {
      this.setError(this.getStreamError(stream))
    }
  })
}
Note that we checked condition if (IS_SAFARI...) and it worked well.

I cannot see any essential difference between our code and Two-way streaming example. Maybe you can? Thank you!
 

Max

Administrator
Staff member
Good day.
Please clarify: can the same mixer output stream on the same device be played in Two Way Streaming example (Player section) or in Player example?
If not, this seems to be media ports blocking issue, according to Failed by ICE timeout error info.
Please check the following:
- if media ports (31001-32000/udp by default) are available on server from the device/network used
- if the stream you're trying to play is available on server
Read here how to check ports availability.
Use /stream/find REST API query to check if stream is published on server:
Code:
POST /rest-api/stream/find HTTP/1.1
Host: wcs:8081
Content-Type: application/json
 
{
    "name":"mixer-stream",
    "published":true
}
 

kindco

Member
Hello @Max ,
The Two-way Streaming example works for mixer streams on iPhone, so it's something with our code.
Do you have any thoughts what can be wrong with it? :)
 

Max

Administrator
Staff member
Do you have any thoughts what can be wrong with it? :)
Please add debug logging to check the flow. Seems like either playFirstVideo and stream playback work in different contexts or playFirstVideo is not invoked at all.
 

Max

Administrator
Staff member
What logging do you mean? Server logs? Some kind of front-end logging?
Frontend logging to the browser console like:
Code:
  if (IS_SAFARI && navigator.mediaDevices || Flashphoner.getMediaProviders()[0] === 'MSE') {
    Flashphoner
      .playFirstVideo(this.mixerRef.current, false, this.preloaderVideo)
      .then(() => {
        console.log("playFirstVideo resolved");
        // Create a mixer stream.
        this.createMixerStream()
        // Play created stream.
        this.mixerStream.play()
      })
      .catch((e) => {
        this.setError(e.message)
        console.error(e)
      })
  }
...
createMixerStream () {
  ...
  // Create a mixer stream.
  // noinspection JSUnresolvedVariable
  console.log("create a stream");
  this.mixerStream = this.wcsSession.createStream({
    name: mixerName + '-' + streamName,
    display: this.mixerRef.current
  })
...
}
 

kindco

Member
Hello @Max

We added console.log() calls almost everywhere:

Code:
/**
 * Creates and plays mixer stream.
 */
playMixerStream () {
  console.log('Entering playMixerStream()')
  console.log('Is Safari: ' + this.isSafari)
  if (Flashphoner.getMediaProviders()[0] === 'WSPlayer') {
    console.log('WSPlayer detected')
    Flashphoner.playFirstSound()
  }
  // This makes everything work in iOS Safari.
  else if (this.isSafari || Flashphoner.getMediaProviders()[0] === 'MSE') {
    console.log('Safari/MSE detected')
    console.log('Playing ' + this.preloaderVideo)
    Flashphoner
      .playFirstVideo(this.mixerRef.current, false, this.preloaderVideo)
      .then(() => {
        console.log('playFirstVideo() resolved')
        // Create a mixer stream.
        this.createMixerStream()
        console.log('About to play')
        // Play created stream.
        this.mixerStream.play()
      })
      .catch((e) => {
        this.setError(e.message)
        console.error(e)
      })
    return
  }

  // Create a mixer stream.
  this.createMixerStream()
  // Play created mixer stream.
  this.mixerStream.play()
}

/**
 * Creates WCS mixer stream.
 */
createMixerStream () {
  const { mixerName, streamName } = this.props
  console.log('About to create a stream')

  // Create a mixer stream.
  // noinspection JSUnresolvedVariable
  this.mixerStream = this.wcsSession.createStream({
    name: mixerName + '-' + streamName,
    display: this.mixerRef.current
  })

  console.log('Stream created')

  // Stream is created and is playing.
  // noinspection JSUnresolvedVariable
  this.mixerStream.on(STREAM_STATUS.PLAYING, () => {
    console.log('Stream playing')
    this.setStatus('live')
  })

  // Stream is failed for some reason.
  this.mixerStream.on(STREAM_STATUS.FAILED, (stream) => {
    console.log('Stream failed')
    const error = this.getStreamError(stream)
    if (error === 'Stopped by publisher stop') {
      this.setStatus(MIXER_POST_ACTION_STATUSES['stop'])
    }
    else {
      this.setError(this.getStreamError(stream))
    }
  })
}
And here is the log we got:
Screen Shot 2021-02-02 at 9.33.19 AM.png

As you can see, playFirstVideo() works, but the stream still fails at the end.
 

kindco

Member
Hello @Max

We added logs for getInfo() and getErrorInfo():
JavaScript:
this.mixerStream.on(STREAM_STATUS.FAILED, (stream) => {
  console.log('Stream failed')
  console.log('getInfo:', stream.getInfo())
  console.log('getErrorInfo:', stream.getErrorInfo())
  const error = this.getStreamError(stream)
  if (error === 'Stopped by publisher stop') {
    this.setStatus(MIXER_POST_ACTION_STATUSES['stop'])
  }
  else {
    this.setError(this.getStreamError(stream))
  }
})
Unfortunately, not too much info here:
Screen Shot 2021-02-03 at 12.38.01 PM.png
 

Max

Administrator
Staff member
Failed by ICE timeout means WebRTC connection cannot be established from the device to the server. It is no depending on client code. So we back to the post above:
- please check if media ports (31001-32000/udp by default) are avalable from the device where error occurs
- please collect a report on server as described here including client debug logs and send using this private form.
 

kindco

Member
Hello @Max
I'm trying to generate a report with sudo ./report.sh --sysinfo --conf --tar, but getting an error Error: Process 2607 does not exist. What can be the issue?
 

Max

Administrator
Staff member
I'm trying to generate a report with sudo ./report.sh --sysinfo --conf --tar, but getting an error Error: Process 2607 does not exist. What can be the issue?
Seems like Java process is stopped. Please check if WCS is running before collecting a report archive. If WCS is running, and issue still persists, please provide SSH access to the server using this private form, we will check.
 

Max

Administrator
Staff member
We've checked you server and found lock file /var/run/FlashphonerWebCallServer.pid from previous installation. Without it, report.sh works correctly (claiming to lsof absence, but this is not mandatory).
But logs are not writing. So we recommend to do the following:
1. Stop WCS.
Code:
systemctl stop webcallserver
2. Clean logs folder
Code:
rm -rf /usr/local/FlashphonerWebCallServer/logs/*
3. Start WCS
Code:
systemctl stop webcallserver
Then, reproduce the "Failed by ICE timeout" issue, collect a report
Code:
sudo ./report.sh --sysinfo --conf --tar
and send using this private form.
 

Max

Administrator
Staff member
Please recollect the logs with INFO log level in log4j.properties file
Code:
log4j.rootLogger=info, stdout, fAppender
Not enough information with WARN level.
 

kindco

Member
Hello,

I tried running ./report.sh --sysinfo --conf --tar from /usr/local/FlashphonerWebCallServer/bin but that script doesn't exist in that location. I wasn't able to find it using "find" either, what is the correct location of this script?
 

Max

Administrator
Staff member
report.sh is in /usr/local/FlashphonerWebCallServer/tools folder:
Code:
cd /usr/local/FlashphonerWebCallServer/tools
sudo ./report.sh --sysinfo --conf --tar
Please see the doc.
 

Max

Administrator
Staff member
We checked the report.
1. About server config
Your server config is too weak for mixing. We now working on mixer load testing and plan to release an article to our blog, please stay tuned. Preliminary results is 2 CPU core per 1 mixer. You're trying to run mixer with default output resolution and bitrate on server instance with 1 CPU core. In addition to mixing, you're republishing output mixer stream to RTMP server which leads to audio transcoding from Opus to AAC. So CPU is probably overloaded.
Also, you have only 2 Gb RAM (1 Gb per Java heap). When mixer is running, garbage collector works every 2 seconds which is too often because every time Java machine stops.
So we recommend you to test on server with at least 2 CPU cores (if only one mixer is used in tests) and 8 Gb RAM (4 Gb per Java heap). The closest config of basic DO droplets is 4 CPU/8 Gb RAM.
2. About server setup
You need to enable multithreaded mixer setting only if you have 10 participants in mixer and more. So you should disable it espacially on weak server.
Also, you're trying to raise publishing bitrate dramatically
Code:
webrtc_cc_min_bitrate = 1500000
webrtc_cc_max_bitrate = 10000000
webrtc_sdp_min_bitrate_bps = 1500000
webrtc_sdp_max_bitrate_bps = 7000000
This may be useful only for 4K streaming through the good channels (20 Mbps). In any other case this may lead to huge packet losses and publishing quality drop (freezes, artifacts, video traffic stopping). We recommend to set lower bitrate boundaries to 100000 bps or leave them as default.
So we recommend to solve the above problems, and then test your code again.
 
Top