Issue with Video chat and Screen sharing (two way communication)

J

jyotipatidar

Member
Hi max,

I need to share both video and screen at the same time. Demo example available with flashphoner server (video chat) is not working as expected.

- Firefox video and audio sharing is not working.
- Chrome screen sharing is not working, (only video sharing works for publisher and participants)

My scenario - First user (Publisher) will share the screen and video, and other users will see it, and all user videos will be visible to the first user.
Video communication should be allowed for all the participants (i.e. Participant should be able to talk to Publisher)

Please suggest if there is any configuration change needed !
 
Max

Max

Administrator
Staff member
We tested Video Chat & Screen Sharing example on our demo server in Chrome and Firefox latest builds. It seems to be working
1594271691304.png

Please note that this example shares screen in Chrome using extension. So to test it on our demo server you should install our extension. To test oun your server, you should either adopt the extension as described here or disable extension by adding the constraint
Code: 
constraints.video.withoutExtension = true
for example
Code: 
function startSharing(room) {
    $("#shareBtn").prop('disabled',true);
    var constraints = {
        video: {
            width: parseInt($('#width').val()),
            height: parseInt($('#height').val()),
            frameRate: parseInt($('#fps').val()),
        },
        audio: $("#useMic").prop('checked')
    };
    constraints.video.type = "screen";
    constraints.video.withoutExtension = true; /// that's the option disabling extension
    if (Browser.isFirefox()){
        constraints.video.mediaSource = "screen";
    }
    room.publish({
        display: document.getElementById("preview"),
        constraints: constraints,
        name: "screenShare",
        cacheLocalResources: false
        ...
    });
}
Also please make sure you're running Firefox and Chrome clients either on different PCs or on the same PC using different cameras (as we've done in our test.
jyotipatidar said:
Also all of sudden disk space use reached100 percent, I have deleted old logs and everything from records folder and report folder , but still not sure what is using 15 GB disk,
Click to expand...
We answered you in this post
 
J

jyotipatidar

Member
we tried your demo page there screen sharing is working

but it's not working on our server

we tried disabling the extension still screen sharing does not work, in the inspect section i see this error
1043Unchecked runtime.lastError: Could not establish connection. Receiving end does not exist.
video-chat-and-screen-sharing.html:1 Unchecked runtime.lastError: Could not establish connection. Receiving end does not exist.

Your immediate reply will be helpful
 
Last edited:
Max

Max

Administrator
Staff member
You should also either prepare your own extension as described here (don't forget to change extension id in example source code) or to remove from example source code any extension checking:
1. Replace the following lines
Code: 
        interval = setInterval(function() {
            chrome.runtime.sendMessage(extensionId, {type: "isInstalled"}, function (response) {
                if (response) {
                    $("#extension").hide();
                    clearInterval(interval);
                    onStopSharing();
                } else {
                    (inIframe()) ? $("#installFromMarket").show() : $("#installExtensionButton").show();
                }
            });
        }, 500);
with
Code: 
        $("#extension").hide();
        clearInterval(interval);
        onStopSharing();
2. Add constraint as described in this post
This should work.
 
J

jyotipatidar

Member
Wee need your input on the followings
1. Stream is not getting saved into records folder
a. We are testing with 3 participants and publisher is sharing his screen, Live is working fine but i do not see the stream in records folder that is saving is not working
I see this error in logs
15:44:23,568 WARN RestUtils - API-ASYNC-pool-12-thread-23 Can't overwrite field timeShift
java.lang.NoSuchFieldException: timeShift
at java.base/java.lang.Class.getDeclaredField(Class.java:2416)
at com.flashphoner.server.commons.util.RestUtils.applyRestOverwriteConfig(Unknown Source)
at com.flashphoner.server.rmi.ManagerApiConnection.getApiMethodResult(Unknown Source)
at com.flashphoner.server.rmi.ManagerApiConnection.lambda$queryApiAsync$0(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1771)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:835)
15:45:14,151 ERROR HttpServerHandler - HTTP-pool-2-thread-120 HTTP error
java.lang.IllegalArgumentException: invalid version format: ="￡ﾘ ﾄﾴ,ﾅﾯN￣YﾻBHL￿(=':ﾩﾂ￙O￈ﾢￗﾓﾘﾴ￯ﾀ￥ﾹﾐ (￀
at org.jboss.netty.handler.codec.http.HttpVersion.<init>(Unknown Source)
at org.jboss.netty.handler.codec.http.HttpVersion.valueOf(Unknown Source)
at org.jboss.netty.handler.codec.http.HttpRequestDecoder.createMessage(Unknown Source)
at org.jboss.netty.handler.codec.http.HttpMessageDecoder.decode(Unknown Source)
at org.jboss.netty.handler.codec.http.HttpMessageDecoder.decode(Unknown Source)
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(Unknown Source)
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(Unknown Source)
at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
at org.jboss.netty.channel.socket.nio.NioWorker.read(Unknown Source)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(Unknown Source)
at org.jboss.netty.channel.socket.nio.DeadlockAwareNioWorker.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

2. Also we want to allow participants to have a conversion with publisher and when he does that
a. Participant video should be highlighted (how we can achieve this ?)
b. Participant audio and video should be available/visible to all the participants (how we can achieve this ?)
 
Max

Max

Administrator
Staff member
jyotipatidar said:
1. Stream is not getting saved into records folder
Click to expand...
Do you enabled recording in stream publishing options as described here? For example
Code: 
    room.publish({
        display: document.getElementById("localDisplay"),
        constraints: constraints,
        record: true, /// This option should be enabled to record stream published
        ...
    });
jyotipatidar said:
I see this error in logs
15:44:23,568 WARN RestUtils - API-ASYNC-pool-12-thread-23 Can't overwrite field timeShift
Click to expand...
This is known issue and does not affect publishing.
jyotipatidar said:
15:45:14,151 ERROR HttpServerHandler - HTTP-pool-2-thread-120 HTTP error
java.lang.IllegalArgumentException: invalid version format:
Click to expand...
This means someone tries to connect to HTTP port using HTTPS or vice versa. Please don't care.
jyotipatidar said:
2. Also we want to allow participants to have a conversion with publisher and when he does that
a. Participant video should be highlighted (how we can achieve this ?)
b. Participant audio and video should be available/visible to all the participants (how we can achieve this ?)
Click to expand...
If you are using RoomApi (this is recommended for small video chats no more then 6 participants simultaneously), every participant stream can be played separately on the same web page. Please rewiew Video Conference and Video Chat and Screen Sharing examples code.
To highlight participant when it speaks you have to impelemnt speech detector. Please look at Media Devices example code, here you can test in on our demo server.
1594349702581.png


You can also use MCU mixer for video chat, test it here. This impelements speaker highlighting, but requires an additional setup and more CPU resources because mixer uses stream transcoding.
 
J

jyotipatidar

Member
ok we will try this out
1) We will have 100 participants joined at a time for video chat and screen sharing (sharing will be allowed only from publisher end), will it support 100 users ?
2) At a time one participant will be talking to publisher and other 99 participant will see the shared screen of the publisher and hear the conversation which is going on , how we can achieve this ?
 
Last edited:
Max

Max

Administrator
Staff member
jyotipatidar said:
We will have 100 participants at a time. That means at a time one participant will be talking to publisher and other 99 participant will see the video (of the one who is speaking) and hear the conversation) , will it support ?
Click to expand...
Please clarify:
1. You mean 100 participants per one chat room?
2. How many video streams should be published simultaneously:
2.1. one from speaker and one from current participant, and these two streams are viewed by other participants
2.2 or every participant publishes its own stream, and everyone should receive all 99 streams?
If yes to 1 and your choice is 2.2, you must use MCU mixer, otherwise it will not work due to channel bandwidth limits of participants.
If your choice is 2.1, you can use RoomApi chat room, but you should implement some logic on frontend, to play only two streams for every client: speaker stream and talking participant stream.
 
J

jyotipatidar

Member
1. You mean 100 participants per one chat room?
Yes

2. How many video streams should be published simultaneously:
I have two scenario -

1. The publishers publish video and share his screen. All 99 participants can watch this streaming (publisher shared screen and publisher video).
Along with this publishers should also be able to view video of all 99 participants

2. Each participant will publish their own stream. And everyone should be able to view all the streams (99 streams).
Along with this Publisher will share two streams (video and screen share)

3. Can we handle both the above case with the same scenario code?

4. When you say MCU requires an additional setup and more CPU resources , can you please provide approx estimate of resources for 100 participants ?
 
Max

Max

Administrator
Staff member
jyotipatidar said:
3. Can we handle both the above case with the same scenario code?
Click to expand...
Yes, using MCU mixer.
Please note that to publish screen share stream you should add the suffix "-desktop" to its name, for example "user1-desktop". In this case, a special layout will be used (screen sharing stream will be placed above the other streams):
1594368547241.png

Anyway, please test MCU Client example on demo server and, after necessary setup, on your server. The example documentatin is here (and source code on GitHub)
jyotipatidar said:
4. When you say MCU requires an additional setup and more CPU resources , can you please provide approx estimate of resources for 100 participants ?
Click to expand...
100 publishers is a good load itself. With mixer, only one encoder will be used, but a much decoders require more memory. So we recommend from 16 CPU cores and from 64 Gb RAM.
 
J

jyotipatidar

Member
For below case also we need MCU mixer ?
1. The publishers publish video and share his screen. All 99 participants can watch this streaming (publisher shared screen and publisher video).
Along with this publishers should also be able to view video of all 99 participants
 
Max

Max

Administrator
Staff member
jyotipatidar said:
For below case also we need MCU mixer ?
1. The publishers publish video and share his screen. All 99 participants can watch this streaming (publisher shared screen and publisher video).
Along with this publishers should also be able to view video of all 99 participants
Click to expand...
Yes. When mixer is used, a publisher (and all the participants) receives one stream. Otherwise, a publisher receives 99 streams. Even with 100 kbps per stream it requires 1 Gbps channel bandwith from publisher to server, so this will not work properly.
Mixer allows to reduce streams count per one client.
 
J

jyotipatidar

Member
Okay, thanks for detailed response, clarification and your immediate response.

We are trying the below implementation with our existing WCS server without MCU for 100 participants
1. The publishers publish video and share his screen. All participants watch this streaming (publisher shared screen and publisher video). Here we have disabled participants video stream.
a. With this only publisher should be able to view video of the participant who is speaking(this is optional if not possible without MCU)
b. rest all participants should be able to hear the conversation of publisher and participant (required only audio stream at all students end)

Challenge we are facing with this implementation is
Participants are unable to here the conversation (voice of the participant who is speaking) (i.e. audio transmission is not working from one participant to all) - can this work without MCU ? If so please suggest sample code
 
Last edited:
Max

Max

Administrator
Staff member
jyotipatidar said:
1. The publishers publish video and share his screen. All participants watch this streaming (publisher shared screen and publisher video). Here we have disabled participants video stream.
a. With this only publisher should be able to view video of the participant who is speaking(this is optional if not possible without MCU)
b. rest all participants should be able to hear the conversation of publisher and participant (required only audio stream at all students end)
Click to expand...
We suppose the following workflow in this case:
1. The speaker publishes video (for example, speakerStream) and share its screen (for example, desktopStream).
2. All participants play speakers streams (speakerStream + desktopStream) with audio and video using constraints constraints: {audio:true, video: true}
3. The participant who wish to speak publishes its video/audio stream (participant1Stream)
4. The speaker plays participant1Stream with audio and video using constraints constraints: {audio:true, video: true}
5. All the rest of participants play participant1Stream with audio only using constraints constraints: {audio:true, video: false}
This case can be implemented without MCU mixer. Every participant will receive 3 streams (2 from speaker and 1 from speaking participant), speaking participant will receive 2 streams from speaker and send 1 stream, the speaker will send 2 streams and receive 1 stream. This seems like acceptable channel and client browser load.
Please inspect Conference example code described here (and source code on GitHub). In this case, all the work should be done on frontend.
If you do not need text chat and any messaging between clients, then RoomApi will be excessive. In this case, you can use Two Way Streaming as an frontend basis, and use REST hooks to impelement backend logic (user authentication, streams naming etc)
 
J

jyotipatidar

Member
Okay thanks for the clarification.

Now on MCU Mixer we need following
1. Participants who is speaking, can we enlarge his video ? how
2. At publisher end Screen Sharing with Participants video visible on his shared screen (may be split shared window and at bottom show participants video
also can we limit only 4 participants visible at a time which are active ?? or randomly pick the video's of participants and show those?
 
Max

Max

Administrator
Staff member
jyotipatidar said:
1. Participants who is speaking, can we enlarge his video ? how
Click to expand...
Using MCU mixer, speaking participant is shown by green frame
1594691732872.png

This is enabled by default. Please see this doc.
jyotipatidar said:
2. At publisher end Screen Sharing with Participants video visible on his shared screen (may be split shared window and at bottom show participants video
Click to expand...
You should use a special suffix for screen sharing stream name (desktop by default) to automatically activate screen sharing layout
1594691956985.png

Please see details here
jyotipatidar said:
also can we limit only 4 participants visible at a time which are active ?? or randomly pick the video's of participants and show those?
Click to expand...
You can use REST API to add/remove participants streams from mixer as you need.
 
J

jyotipatidar

Member
1. speaking participant is shown by green frame but how we can enlarge his/her video when it is green ?

2. And we already tried the option of desktop which you suggested
My issue is when Publisher shares his screen he is not able to see the participants on the shared window, is it possible to show a spited shared window at publisher end also ? where after sharing the screen publisher should also be able to view the participants ?
 
You must log in or register to reply here.
Top