Filters (beautify, AR, etc) on live streaming

andrew.n

andrew.n

Member
@Max
Everything works great with the SDK, and of course, new requests came up such as using filters during the live stream.
I want to ask you if you had that kind of request before, or if there is another SDK to add over the current FlashPhoner SDK to handle this request.
It will be useful if the user can set up the beautify to select what happens with the skin, eyes, mouth, nose etc, but also it will be even better if the user will be able to set custom objects predefined, same as you can do on Snapchat.
At this point we need to know what options we have for all platforms (web, iOS, Android).
 
andrew.n

andrew.n

Member
@Max I started to look into the project, but I want to ask, is there any documentation regarding all filters? Like a preview for us to see how all of them look like without having to test each of them ?
Also, is there any guide how to create your own filter using your SDK ?
 
Max

Max

Administrator
Staff member
andrew.n said:
I started to look into the project, but I want to ask, is there any documentation regarding all filters? Like a preview for us to see how all of them look like without having to test each of them ?
Click to expand...
Filters preview is available in example applications, so you have to build them to test.
andrew.n said:
Also, is there any guide how to create your own filter using your SDK ?
Click to expand...
In Web SDK, all filters code seems trivial: GitHub
In Android SDK and iOS SDK, GPU Image library is used, so you can refer to its documentation to implement a custom filters. Note that GPU Image has a perfomance issues on Android devices.
 
Max

Max

Administrator
Staff member
Hello

We have customers who do similar things such as face detection, beautification, etc.
We have implemented support of GPU Image Library for this case.
However we didn't use any filters just very simple custom filters for demonstration purposes.

You would need to search more about GPUImage

Example GPUImage Bulge Distortion Filter and Face Detection

Hi all This 2 small Example First example GPUImage Bulge Distortion Filter and Face Detection next Filter and Face Detection maybe Weak simple example It enlarges part of the image, suitable for making facial, eye and nose expressions First example need add GPUImageFramework. next example...
www.b4x.com www.b4x.com
 
Max

Max

Administrator
Staff member
We support both Objective С and Swift, but GPUImage3 is Swift only framework. So we have to stay with GPUImage1.
 
Max

Max

Administrator
Staff member
GPUImage is used as CocoaPods dependency in GPU Image Swift example. So you can try to set the appropriate version if it is present in CocoaPods.
 
andrew.n

andrew.n

Member
@Max I'm playing around with the GPUImageDemoViewController. It worked to start the stream and play it (it uses the back camera) and I added another button to switch the camera:
Code: 
@IBAction func didTapSwitchButton(_ sender: UIButton) {
   publishStream!.switchCamera()
}
The camera is not switching but on the debugger I see the following:
Code: 
2022-08-29 12:21:29.761046+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Finded device: Back Camera
2022-08-29 12:21:29.761139+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try to find dimension: 640 480
2022-08-29 12:21:29.761177+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 192 144
2022-08-29 12:21:29.761206+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 192 144
2022-08-29 12:21:29.761234+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 352 288
2022-08-29 12:21:29.761261+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 352 288
2022-08-29 12:21:29.761287+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 480 360
2022-08-29 12:21:29.761313+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 480 360
2022-08-29 12:21:29.761338+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 640 480
2022-08-29 12:21:29.761471+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 640 480
2022-08-29 12:21:29.761503+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 640 480
2022-08-29 12:21:29.761528+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 640 480
2022-08-29 12:21:29.761548+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 960 540
2022-08-29 12:21:29.761559+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 960 540
2022-08-29 12:21:29.761569+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1024 768
2022-08-29 12:21:29.761580+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1024 768
2022-08-29 12:21:29.761590+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1280 720
2022-08-29 12:21:29.761934+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1280 720
2022-08-29 12:21:29.761953+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1280 720
2022-08-29 12:21:29.761963+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1280 720
2022-08-29 12:21:29.761974+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1280 720
2022-08-29 12:21:29.761985+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1280 720
2022-08-29 12:21:29.761995+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1280 720
2022-08-29 12:21:29.762005+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1280 720
2022-08-29 12:21:29.762015+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1440 1080
2022-08-29 12:21:29.762026+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1440 1080
2022-08-29 12:21:29.762893+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1920 1080
2022-08-29 12:21:29.762921+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1920 1080
2022-08-29 12:21:29.762935+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1920 1080
2022-08-29 12:21:29.762946+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1920 1080
2022-08-29 12:21:29.762957+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1920 1080
2022-08-29 12:21:29.762968+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1920 1080
2022-08-29 12:21:29.762979+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1920 1080
2022-08-29 12:21:29.762989+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1920 1080
2022-08-29 12:21:29.762999+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1920 1080
2022-08-29 12:21:29.763333+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1920 1080
2022-08-29 12:21:29.763411+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1920 1440
2022-08-29 12:21:29.763432+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1920 1440
2022-08-29 12:21:29.763444+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1920 1440
2022-08-29 12:21:29.763458+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1920 1440
2022-08-29 12:21:29.763469+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 2592 1944
2022-08-29 12:21:29.763479+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 2592 1944
2022-08-29 12:21:29.763490+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 3264 2448
2022-08-29 12:21:29.763686+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 3264 2448
2022-08-29 12:21:29.763712+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 3840 2160
2022-08-29 12:21:29.763724+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 3840 2160
2022-08-29 12:21:29.763736+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 3840 2160
2022-08-29 12:21:29.763746+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 3840 2160
2022-08-29 12:21:29.763756+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 4032 3024
2022-08-29 12:21:29.763766+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 4032 3024
2022-08-29 12:21:29.763819+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Selected dimension: 640 480 420f
2022-08-29 12:21:29.763838+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Selected fps: 30
 
andrew.n

andrew.n

Member
If I press again the first line in the debugger changes:
Code: 
[CODE]2022-08-29 12:21:51.848065+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Finded device: Front Camera
2022-08-29 12:21:51.848145+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try to find dimension: 640 480
2022-08-29 12:21:51.848172+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 192 144
2022-08-29 12:21:51.848192+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 192 144
2022-08-29 12:21:51.848210+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 352 288
2022-08-29 12:21:51.848227+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 352 288
2022-08-29 12:21:51.848245+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 480 360
2022-08-29 12:21:51.848262+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 480 360
2022-08-29 12:21:51.848279+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 640 480
2022-08-29 12:21:51.848741+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 640 480
2022-08-29 12:21:51.848778+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 640 480
2022-08-29 12:21:51.848797+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 640 480
2022-08-29 12:21:51.848816+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 960 540
2022-08-29 12:21:51.848833+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 960 540
2022-08-29 12:21:51.848850+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1024 768
2022-08-29 12:21:51.848868+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1024 768
2022-08-29 12:21:51.848886+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1280 720
2022-08-29 12:21:51.849023+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1280 720
2022-08-29 12:21:51.849044+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1280 720
2022-08-29 12:21:51.849061+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1280 720
2022-08-29 12:21:51.849079+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1280 720
2022-08-29 12:21:51.849096+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1280 720
2022-08-29 12:21:51.849112+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1280 720
2022-08-29 12:21:51.849129+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1280 720
2022-08-29 12:21:51.849146+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1440 1080
2022-08-29 12:21:51.849163+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1440 1080
2022-08-29 12:21:51.849513+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1920 1080
2022-08-29 12:21:51.849547+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1920 1080
2022-08-29 12:21:51.849567+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1920 1080
2022-08-29 12:21:51.849586+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1920 1080
2022-08-29 12:21:51.849603+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1920 1080
2022-08-29 12:21:51.849621+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1920 1080
2022-08-29 12:21:51.849638+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1920 1080
2022-08-29 12:21:51.849655+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1920 1080
2022-08-29 12:21:51.851111+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1920 1440
2022-08-29 12:21:51.851151+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1920 1440
2022-08-29 12:21:51.851169+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1920 1440
2022-08-29 12:21:51.851187+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 1920 1440
2022-08-29 12:21:51.851406+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 3088 2316
2022-08-29 12:21:51.851566+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 3088 2316
2022-08-29 12:21:51.851582+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 3840 2160
2022-08-29 12:21:51.851595+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 3840 2160
2022-08-29 12:21:51.851608+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 3840 2160
2022-08-29 12:21:51.851966+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 3840 2160
2022-08-29 12:21:51.851993+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 4032 3024
2022-08-29 12:21:51.852010+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Try dimension: 4032 3024
2022-08-29 12:21:51.852098+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Selected dimension: 640 480 420f
2022-08-29 12:21:51.852129+0300 GPUImageDemoSwift[66565:4778931] [FPWCSApi2WebRTCMediaProvider] Selected fps: 30
[/CODE]
 
andrew.n

andrew.n

Member
Max said:
We support both Objective С and Swift, but GPUImage3 is Swift only framework. So we have to stay with GPUImage1.
Click to expand...
A small note on this, it worked to include GPUImage v3 in the project. The project had to be added manually because it's not available with CocoaPods, so firstly, you have to remove the v1 from CocoaPods and add the GPUImage v3 as a manual project into your project.
 
Max

Max

Administrator
Staff member
andrew.n said:
The camera is not switching but on the debugger I see the following:
Click to expand...
Please make sure the stream is publishing when you're trying to switch the camera. If yes, please y=try to reproduce the issue in Two Way Streaming Swift example. Then modify example souce code minimally to reproduce the issue and send us the code using this form.
andrew.n said:
A small note on this, it worked to include GPUImage v3 in the project. The project had to be added manually because it's not available with CocoaPods, so firstly, you have to remove the v1 from CocoaPods and add the GPUImage v3 as a manual project into your project.
Click to expand...
We will add this to documentation, thanks for testing.
 
Max

Max

Administrator
Staff member
We reproduced the problem in GPUImageDemo Swift example and raised the ticket WCS-3584. Will let you know about progress.
 
You must log in or register to reply here.
Top