Issue with Video chat and Screen sharing (two way communication)

[Max]

1. speaking participant is shown by green frame but how we can enlarge his/her video when it is green ?

This cannot be done. Only screen sharing stream can be enlarged.

2. And we already tried the option of desktop which you suggested
My issue is when Publisher shares his screen he is not able to see the participants on the shared window, is it possible to show a spited shared window at publisher end also ? where after sharing the screen publisher should also be able to view the participants ?

Please test MCU Client example on your server https://wcs:8444/client2/examples/demo/streaming/mcu_client/mcu_client.html or on our demo server https://demo.flashphoner.com:8444/client2/examples/demo/streaming/mcu_client/mcu_client.html
Every participant, including the Publisher, should play mixer output stream to see all the other participants.

 

[jyotipatidar]

I think i was not able to clarify the second point
Let me rephrase ,
with the MCU mixer solution I can see publisher with splitted view but when we add sharing (desktop prefix) publisher only see shared window without participants video. (after enabling screen share, at publisher end splitting view is not coming, he can only see his shared window)

 

[Max]

with the MCU mixer solution I can see publisher with splitted view but when we add sharing (desktop prefix) publisher only see shared window without participants video. (after enabling screen share, at publisher end splitting view is not coming, he can only see his shared window)

Do you add screen sharing stream to mixer? If not, it should be done with REST API /mixer/add or while creating a stream by addin mixer namewith a special suffix, for example desktop_stream#mixer1. In this case, mixer desktop layout will be enabled automatically.
To test, you can publish a stream even from web camera with stream name formed as above, the stream with such name will be added as screen share.

 

[jyotipatidar]

Hi Max,
I have attached the pictorial representation of what we need at Publisher end. Please refer it and suggest.

Also how can we add Joined Participant name on his/her video ?

 

[Max]

I have attached the pictorial representation of what we need at Publisher end. Please refer it and suggest.

MCU view with screen sharing is the same for all participants including publisher. Publisher can attach to PC and share a second screen if you do not wish to see its own browser page.

Also how can we add Joined Participant name on his/her video ?

Enable the following option

mixer_display_stream_name=true

and name streams as users, for example user1#room1

 

[jyotipatidar]

1. Ok, it's working now, but can we remove room number from showing it from video caption ?

2. We need to also save mixed stream (MCU)
We tried the following but it is only saving publisher video.
room.publish({
display: document.getElementById("localDisplay"),
constraints: constraints,
record: true, /// This option should be enabled to record stream published
...
});

We need to save the communication between pubisher and participant and publisher shared screeen. That means the stream which is playing at participants end needs to be saved.

3. On Video chat and Sharing also how do we save Publisher video/shared screen and the communication witth all the participants ? (mean the real session should save as is with shared screen and communication and participants video/audio - same as participant view)

 

[Max]

1. Ok, it's working now, but can we remove room number from showing it from video caption ?

To remove the room name from stream name, you should:
1. Publish a participant stream with participant name only, for example user1
2. For the first participant, create mixer using REST API query /mixer/startup

{
    "uri": "mixer://room1",
    "localStreamName": "room1"
}

3. Add the participant stream to the mixer using REST API query /mixer/add

{
    "uri": "mixer://room1",
    "remoteStreamName": "user1"
}

In this case, only stream name will be shown in mixer picture caption

2. We need to also save mixed stream (MCU)

You can start mixer output stream recording using REST API by the following way:
1. Find mixer stream by name using REST API query /stream/find

{
    "name":"room1",
    "published":true
}

and get its mediaSessionId, for example

[
    {
        "custom":{},
        "nodeId":null,
        "appKey":"defaultApp",
        "sessionId":"/192.168.1.102:34500/192.168.1.101:8080",
        "mediaSessionId":"4f112b20-13d0-11e7-b521-59a9cb7eddeb",
        ...
    }
]

2. Start mixer stream recording with REST API query /stream/startRecording

{
    "mediaSessionId":"4f112b20-13d0-11e7-b521-59a9cb7eddeb"
}

3. Optionally, stop mixer stream recording after the conference is finished using REST API query /stream/stopRecording

{
    "mediaSessionId":"4f112b20-13d0-11e7-b521-59a9cb7eddeb"
}

3. On Video chat and Sharing also how do we save Publisher video/shared screen and the communication witth all the participants ? (mean the real session should save as is with shared screen and communication and participants video/audio - same as participant view)

The best way is to use mixer for conference recording. Please note that all the layout is defined on client side in this case, so mixer desktop layout will be used to record, if screen share stream is present, or grid layout, if not.

 

[jyotipatidar]

Thanks Max for solving all the issues

Now we are using custom layout as given here
1. I see that only 4 participants are added after implementing left grid, can we remove number of participants restriction in this custom layout ?
may be we can switch the participants randomly in this view ?

2. In this TestLayout can we enable default as grid layout if publisher is not sharing screen or stops sharing the screen ? if so how ?

 

[Max]

1. I see that only 4 participants are added after implementing left grid, can we remove number of participants restriction in this custom layout ?
may be we can switch the participants randomly in this view ?

Please read this and this posts for description how to fix the issue

2. In this TestLayout can we enable default as grid layout if publisher is not sharing screen or stops sharing the screen ? if so how ?

You should rework the code:
- check if there is a desktop stream in stream names list
- if yes, implement desktop layout as described in that topic
- if no, implement grid layout as described in the doc
If this seems too hard, please wait for desktop layout customization feature on which we work in ticket WCS-2752

 

[jyotipatidar]

1. Please read this and this posts for description how to fix the issue
I have implemented this but it's allowing only 4 participants to enter the room

 

[Max]

I have implemented this but it's allowing only 4 participants to enter the room

From this post:

The reason is this string

layoutHeight = Math.min((int) ((layoutWidth*9.0d/16.0d)), (int) ((canvasHeight - (yUVFrameArr.length - 1) * PADDING_IN)/ (double) (yUVFrameArr.length - 1)));

In this expression (yUVFrameArr.length - 1) * PADDING_IN)/ (double) (yUVFrameArr.length - 1), yUVFrameArr.length is actually shortened, and picture height is always canvasHeight - PADDING_IN. So 5th stream picture goes out of main picture height.
Please refactor this line to

layoutHeight = Math.min((int) ((layoutWidth*9.0d/16.0d)), (int) ((canvasHeight / yUVFrameArr.length) - PADDING_IN));

for example. With this fix, server is not crashed in our test environment.

If this seems too hard, please wait for desktop layout customization feature on which we work in ticket WCS-2752. We'll let you know when add the feature.

 

[jyotipatidar]

From this post:

If this seems too hard, please wait for desktop layout customization feature on which we work in ticket WCS-2752. We'll let you know when add the feature.

Any estimate date by when ticket WCS-2752 fix will be available for us ?

 

[Max]

Any estimate date by when ticket WCS-2752 fix will be available for us ?

We do not provide ETA for new features, but this feature is moved up because we already have at list two user votes for it. Please be patient.

 

[jyotipatidar]

Okay, do let us know when it is available, we will be waiting
Also if you can add below features, it will be helpful
1. Default as grid Layout and after sharing screen enabled left grid Layout view
2. Show Publisher left side and other participant right side (just to differentiate who is publisher) or configurable placement of videos
3. Option to enlarge publisher and speaking participant video
4. Show random videos of all participants ( may be configurable as needed) show most active on screen rest all participants videos can rotate at a fix interval
5. Option to display room session videos on full screen

These are just suggestions if we can have will add value !

 

[Max]

1. Default as grid Layout and after sharing screen enabled left grid Layout view

This is the default behaviour. Desktop layout will be configurable (left, right, top, bottom grid)

2. Show Publisher left side and other participant right side (just to differentiate who is publisher) or configurable placement of videos

This can be achieved using custom layout.

3. Option to enlarge publisher and speaking participant video

This can be (visually) achieved by increasingreen frame thickness

mixer_voice_activity_frame_thickness=10

4. Show random videos of all participants ( may be configurable as needed) show most active on screen rest all participants videos can rotate at a fix interval

There is no criteria on server side to determine a most active participant. You can try to implement this using custom layout based on stream names.

5. Option to display room session videos on full screen

Mixer output stream can be played in full screen on browser side. Please check Player example source code.

 

[Max]

Good day.
Since build 5.2.710 it is possible to change screen sharing picture placement with the following parameter:

mixer_desktop_align=RIGHT

Please read the details here.

 

[jyotipatidar]

Thank you so much, we will test and let you know incase any further issue

 

[jyotipatidar]

Hi Max,
This new layout implementation just works great but we have some custom requirement.
1. Is it possible to share the RIGHT and LEFT layout java files with us ?

2. Also how do we debug this custom java file changes ? i want to see console log of this TestLayout.java

 

[Max]

Good day.

1. Is it possible to share the RIGHT and LEFT layout java files with us ?

No, it is impossible. You should implement custom layout if standard layout is not enough.

2. Also how do we debug this custom java file changes ? i want to see console log of this TestLayout.java

You can use log4j logger to debug as folows
1. Add package import to custom layout soure file

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

2. Get the logger object

/**
* Custom mixer layout implementation example
*/
public class TestLayout implements IVideoMixerLayout {

    private static final Logger log = LoggerFactory.getLogger("CustomLayout");
    ...

3. Log all what you need to DEBUG level

    @Override
    public IVideoMixerLayout.Layout[] computeLayout(YUVFrame[] yUVFrameArr, String[] strArr, int canvasWidth, int canvasHeight) {

        log.debug("Custom mixer layout will place " + yUVFrameArr.length + " pictures to mixer canvas");
        ...
        // Return pictures layouts calculated as array
       return layout.toArray(new IVideoMixerLayout.Layout[layout.size()]);
    }
}

4. Compile custom layuot class

 javac -cp "/usr/local/FlashphonerWebCallServer/lib/slf4j-api-1.6.4.jar:/usr/local/FlashphonerWebCallServer/lib/wcs-core.jar" ./com/flashphoner/mixerlayout/TestLayout.java
jar -cf testlayout.jar ./com/flashphoner/mixerlayout/TestLayout.class
cp testlayout.jar /usr/local/FlashphonerWebCallServer/lib

5. Enable custom layout in server settings

mixer_layout_class=com.flashphoner.mixerlayout.TestLayout

6. Enable custom layout debug log in file log4j.properties

log4j.logger.CustomLayout=DEBUG

7. Restart server
Then, starting a mixer, you will see in server log file your debug info, for example

13:49:32,454 INFO            MixerAgent - API-ASYNC-pool-12-thread-1 Agent mixer://m1-40ffb37e-f429-4898-9c0f-54f208b8a0b3 changed state to PROCESSED_LOCAL
13:49:32,461 INFO            MixerAgent - AGENT-PLAY-STREAM-pool-42-thread-1 Found session with stream name test#m1, adding to mix
13:49:33,564 DEBUG         CustomLayout - MIXER-AGENT-mixer://m1-40ffb37e-f429-4898-9c0f-54f208b8a0b3 Custom mixer layout will place 1 pictures to mixer canvas
13:49:33,603 DEBUG         CustomLayout - MIXER-AGENT-mixer://m1-40ffb37e-f429-4898-9c0f-54f208b8a0b3 Custom mixer layout will place 1 pictures to mixer canvas
...

Please note that this debug output can take a much disk space and can affect mixer perfomance because custom mixer layout is invoked per every mixer frame. So it if not recommended to leave this output in production.