WCS5.1 on ubuntu 18.04.1 LTS License Activation Issue

[Stanley]

Hi Max,

WCS5.1 keep prompting license activation when I access admin page whenever I perform "service webcallserver restart" or "service webcallserver stop then start" command. No issue after rebooted the server.

I've confirm that my server able to resolved "localhost" to 127.0.0.1. Server able to resolve "my.flashphoner.com" for license checking.

Any clue?

 

[Max]

Hello,

No issue after rebooted the server.

Please clarify – the issue is not reproducible anymore?
Here is a similar thread.
What URL is used to access the dashboard?
Does the server hostname resolve to IP?

 

[Stanley]

Please clarify – the issue is not reproducible anymore?
No matter how I have to reboot the server instead of restarting service.

What URL is used to access the dashboard?
http://{IP}:9091 --- with -Dserver.port=9091 enabled in flashphoner.properties

Does the server hostname resolve to IP?
Able to resolve all IPs. localhost, myhostname, external dns & etc...

 

[Max]

Please clarify again:
1. You boot your Ubuntu machine.
2. You start WSC with

service webcallserver start

3. WCS starts as described here, you can see 2 processes (WCS core and manager)
If you try to open WCS web interface https://host:8888, what happens on this step?
4. You restart WCS with

service webcallserver restart

5. WCS starts as described here, you can see 2 processes (WCS core and manager)
If you try to open WCS web interface https://host:8888, what happens on this step?

Then, if WCS is not started on step 3:
1. Did you try to activate license with command line as described here?
2. Are core and manager logs written? What is displayed in logs?

If WCS started successfully, and license activated successfully, but WCS prompts license activation each time you login to web interface:
1. Is there enough space on server hard drive?
2. Did you try to completely uninstall WCS

/usr/local/FlashphonerWebCallServer/bin/uninstall.sh

and install it again?

 

[Stanley]

Please clarify again:
1. You boot your Ubuntu machine.
2. You start WSC with

service webcallserver start

3. WCS starts as described here, you can see 2 processes (WCS core and manager)
If you try to open WCS web interface https://host:8888, what happens on this step?
4. You restart WCS with

service webcallserver restart

5. WCS starts as described here, you can see 2 processes (WCS core and manager)
If you try to open WCS web interface https://host:8888, what happens on this step?

Then, if WCS is not started on step 3:
1. Did you try to activate license with command line as described here?
2. Are core and manager logs written? What is displayed in logs?

If WCS started successfully, and license activated successfully, but WCS prompts license activation each time you login to web interface:
1. Is there enough space on server hard drive?
2. Did you try to completely uninstall WCS

/usr/local/FlashphonerWebCallServer/bin/uninstall.sh

and install it again?

Please clarify again:
1. You boot your Ubuntu machine.
2. You start WSC with

service webcallserver start

3. WCS starts as described here, you can see 2 processes (WCS core and manager)
If you try to open WCS web interface https://host:8888, what happens on this step?
4. You restart WCS with

service webcallserver restart

5. WCS starts as described here, you can see 2 processes (WCS core and manager)
If you try to open WCS web interface https://host:8888, what happens on this step?

WCS is auto startup after boot the machine. Everything is normal when the system is fresh booted. I am able to login

http://WCSIP:9091

into admin page. I'm able to browse

http://WCSIP:8081/?action=stat

everything is working as expected.

Here is the scenario....
Only when I run

service webcallserver restart

command, WCS admin page will prompt for license activation. I immediately run ./activation in /bin file

root@prd-wcs03:/usr/local/FlashphonerWebCallServer/bin# service webcallserver stop
root@prd-wcs03:/usr/local/FlashphonerWebCallServer/bin# service webcallserver start

WCS failed... :9091 admin page prompt for license activation... then i execute below

root@prd-wcs03:/usr/local/FlashphonerWebCallServer/bin# ./activation.sh
------------------------------
Flashphoner License Activation
------------------------------

Do you agree deactivate current license (17FXXXXX-0692-XXXX-XX83-XXXXXXXXXXXX)? [yes/no]

root@prd-wcs03:/usr/local/FlashphonerWebCallServer/bin# ps aux | grep WebCallServer
root     18229 53.9  4.2 4320528 1384684 ?     Sl   12:14   3:03 java -Dloader.path=/usr/local/FlashphonerWebCallServer-5.1.3511-ebcbd3628aba5ea472cb91e6ae3d17e1a3e7efa7/lib/tbs-commons.jar,/usr/local/FlashphonerWebCallServer-5.1.3511-ebcbd3628aba5ea472cb91e6ae3d17e1a3e7efa7/lib/wcs_manager-1.0.jar -Dcom.flashphoner.fms.AppHome=/usr/local/FlashphonerWebCallServer -Dnode.enable_stdout=false -Dserver.port=9091 -Drest_template.user_agent=WCS5 -Dnode.stdout_count_files=5 -Dnode.stdout_max_file_size=1000000 -Dstream_stat_persist_data=true -Dspring.profiles.active=production -Xmx2048M -XX:+PrintGCDateStamps -XX:+PrintGCDetails -Xloggc:/usr/local/FlashphonerWebCallServer/logs/gc-manager-2018-10-05_12-14.log -Djava.net.preferIPv4Stack=true -XX:+PrintGCDateStamps -jar /usr/local/FlashphonerWebCallServer-5.1.3511-ebcbd3628aba5ea472cb91e6ae3d17e1a3e7efa7/lib/wcs_manager-1.0.jar -Xms16g -Xmx16g -XX:+UseConcMarkSweepGC -XX:NewSize=2048m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=50999 -Djava.rmi.server.hostname=localhost -XX:ErrorFile=/usr/local/FlashphonerWebCallServer/logs/error%p.log -XX:+PrintGCDateStamps -XX:+PrintGCDetails -Xloggc:/usr/local/FlashphonerWebCallServer/logs/gc-core-2018-10-05_12-14.log -XX:+ExplicitGCInvokesConcurrent -Dsun.rmi.dgc.client.gcInterval=36000000000 -Dsun.rmi.dgc.server.gcInterval=36000000000 -Dcom.flashphoner.fms.AppHome=/usr/local/FlashphonerWebCallServer -Djava.library.path=/usr/local/FlashphonerWebCallServer/lib/so:/usr/local/FlashphonerWebCallServer/lib -DWCS_NON_ROOT=false -DsessionDebugEnabled=false -Djdk.tls.client.protocols="TLSv1,TLSv1.1,TLSv1.2" -cp /usr/local/FlashphonerWebCallServer/lib/* com.flashphoner.server.Server
root     18597  0.0  0.0  13136  1012 pts/0    S+   12:20   0:00 grep --color=auto WebCallServer
root@prd-wcs03:/usr/local/FlashphonerWebCallServer/bin#

I've emailed you my server logs and conf file. This issue happened in 2 x servers.

 

[Max]

You have too old version. It is strongly recommended to download latest version from this page and make clean installation (uninstall current version and install newest).
In your WCS manager log files (you didn't sent us WCS core logs that should be created at WCS_HOME/logs/server_logs) we see

12:22:49,869 ERROR Http11NioProtocol - main Failed to start end point associated with ProtocolHandler ["http-nio-8888"]
java.io.IOException: keystore password was incorrect

In means that your WCS keystore file with SSL certificate WCS_HOME/conf/wss.jks probably was lost or unreadable. In conf directory that you sent us this file is absent.
So, you have to:
1. Download latest WCS version from this page
2. Uninstall your current WCS installation with command

/usr/local/FlashphonerWebCallServer/bin/uninstall.sh

3. Install latest version
4. Activate it with command line as described here
5. Set up SSL certificate as described here
If after all of those steps the issue persists, provide us SSH access to your server, and we will check.

 

[Stanley]

You have too old version. It is strongly recommended to download latest version from this page and make clean installation (uninstall current version and install newest).
In your WCS manager log files (you didn't sent us WCS core logs that should be created at WCS_HOME/logs/server_logs) we see

In means that your WCS keystore file with SSL certificate WCS_HOME/conf/wss.jks probably was lost or unreadable. In conf directory that you sent us this file is absent.
So, you have to:
1. Download latest WCS version from this page
2. Uninstall your current WCS installation with command

/usr/local/FlashphonerWebCallServer/bin/uninstall.sh

3. Install latest version
4. Activate it with command line as described here
5. Set up SSL certificate as described here
If after all of those steps the issue persists, provide us SSH access to your server, and we will check.

1. Download latest WCS version from this page

Will try on this. Another issue is when installing WCS the folder name weirdly and unable to fully uninstall. I have to manually remove the folder.

bin  etc  FlashphonerWebCallServer  FlashphonerWebCallServer-5.1.3511-ebcbd3628aba5ea472cb91e6ae3d17e1a3e7efa7  games  include  lib  man  sbin  share  src

In means that your WCS keystore file with SSL certificate WCS_HOME/conf/wss.jks probably was lost or unreadable. In conf directory that you sent us this file is absent.

Because I didn't send this file to you. As mentioned, it is working perfectly when the system fresh boot.

 

[Max]

when installing WCS the folder name weirdly and unable to fully uninstall

There is no issue with the directory name. If the config files you sent are from this server when 5.1.3511 was already installed, there should have been 5.1.3511-ebcbd3628aba5ea472cb91e6ae3d17e1a3e7efa7 in the WCS.version file yet there is 5.1.3253. Then that is why the directory was not removed: its name didn't correspond to the version in WCS.version.

 

[Stanley]

Max, license issue resolved after updating to v5.1.3578 but REST hooks to our API server with HTTPS is not working. No issue with HTTP.

 

[Max]

Hello, Stanley.

REST hooks to our API server with HTTPS is not working. No issue with HTTP

Please check if url of your REST hook application matches url that confirmed by SSL certificate. For example, if we have SSL certificate on *.domain.com, and WCS tries to make query to https://localhost:8444/, it will fail with this message in WCS_HOME/logs/server_logs/flashphoner.log

17:50:34,861 ERROR RestClient - main Got exception in REST
javax.net.ssl.SSLException: hostname in certificate didn't match: <localhost> != <*.domain.com> OR <*.domain.com> OR <domain.com>

You can check and change application urls via WCS Core CLI or by editing file WCS_HOME/conf/database.yml manually.

 

[Stanley]

WCS able to access to our API but server responded Forbidden causing player failed to stream. Is the REST hoom Type 1 json format still the same in latest WCS version?

 

[Stanley]

I also noticed that my previous version v5.1.3511 REST hook actually not working too. It didnt call my API server at all using custom app key and simply allow player to stream.

https://docs.flashphoner.com/display/WCS5EN/Type+1+-+the+connect+method
this documentation seems not updated. The default is still usinf http://localhost:9091/EchoApps instead of http://localhost:8081/apps/EchoApps

 

[Max]

In cleanly installed 5.1.3578, the default URL for e.g. defaultApp is 'http://localhost:8081/apps/EchoApp'. In case of update to 5.1.3578 it also should have been changed to that. Where do you see the URLs for apps? CLI on port 2001, or database.yml should be used. Also, please verify that required methods (e.g., ConnectionStatusEvent) are added to the custom application.

 

[Stanley]

Max, license issue resolved after updating to v5.1.3578 but REST hooks to our API server with HTTPS is not working. No issue with HTTP.

Hi Max,
I discover something serious in v5.1.3578. The rest hook will stop working after several minutes WCS started. We are using

https://docs.flashphoner.com/display/WCS5EN/Type+1+-+the+connect+method

REST Hooks Type-1 method to authenticate players.

Here is the scenario.
1. Player request token from API server
2. API Server generate and response example {Token1} (token valid for 1 minute stored in server) to player
3. Player send {Token1} + video + appkey & etc request to Flashphoner
4. Flashphoner based on appkey send custom: {Token1} to API server
5. API server check Token1 and response with HTTP 200 to Flashphoner
6. Flashphoner allow stream to player

After several minutes of WCS started and concurrent connection about 30 users. Flashphoner will delay sending REST Hook request to API server. The delay was about 5 to 20 minutes where the Token1 already expired in API server.

 

[Max]

Please send server and client logs (WCS_HOME/logs/server_logs, WCS_HOME/logs/client_logs) collected when it happens, we will check.

 

[Max]

Please prepare a full report as described here and send us. No any parts should be excluded. We will check.

 

[Stanley]

Please prepare a full report as described here and send us. No any parts should be excluded. We will check.

Hi Max, I've sent the conf and logs to support@flashphoner.com please take note do not post any domain and any sensitive info into the forum.

Thank you.

 

[Max]

Hello.
In your logs we see two issues (all sensitive info is hidden as you wish):
1. It seems like players may connect and disconnect too fast, for example

21:50:30,863 INFO WSServerHandler - WSS-pool-21-thread-4 Orgign: http://yourclient
...
21:50:31,085 INFO RestClient - API-ASYNC-pool-12-thread-5 SEND REST OBJECT ==>
URL:https://yourbackend/connect
OBJECT:
{
...
"sessionId" : "/source_ip:source_port/server_ip:443",
...
"origin" : "http://yourclient"
}
...
21:50:39,356 INFO WSClient - WSS-pool-21-thread-3 Close connection for channel [id: 0x08ed1ec5, /source_ip:source_port => /server_ip:443] with status code: 1000 ; reason: Normal disconnect

In this case, client closed connection before your backend server replied to /connect query. It's not a big problem, but
2. It seem like your backend server may reply with big delay, for example

21:50:30,863 INFO WSServerHandler - WSS-pool-21-thread-4 Orgign: http://yourclient
...
21:50:31,085 INFO RestClient - API-ASYNC-pool-12-thread-5 SEND REST OBJECT ==>
URL:https://yourbackend/connect
OBJECT:
{
...
"sessionId" : "/source_ip:source_port/server_ip:443",
...
"origin" : "http://yourclient"
}
...
21:50:56,098 WARN RestClient - API-ASYNC-pool-12-thread-5 POST request resulted in 403 (Forbidden)
21:50:56,098 WARN ManagerApiConnection - API-ASYNC-pool-12-thread-5 Failed to get object from REST with exception:Forbidden
...
21:50:56,099 INFO WSClient - API-ASYNC-pool-12-thread-5 Close connection for channel [id: 0x08ed1ec5, /source_ip:source_port => /server_ip:443] with status code: 1000 ; reason: Disconnected by error on REST method /connect

It is the same case as above, and your backend server replied after 25 seconds delay.
Probably you should check your backend. And we strongly recommend to implement REST hooks to return exactly the same body as received (mirroring). It may help to debug such issues. The /connect REST hook implementation example for users authentication by domain is described here.

 

[Stanley]

Hello.
In your logs we see two issues (all sensitive info is hidden as you wish):
1. It seems like players may connect and disconnect too fast, for example

In this case, client closed connection before your backend server replied to /connect query. It's not a big problem, but
2. It seem like your backend server may reply with big delay, for example

It is the same case as above, and your backend server replied after 25 seconds delay.
Probably you should check your backend. And we strongly recommend to implement REST hooks to return exactly the same body as received (mirroring). It may help to debug such issues. The /connect REST hook implementation example for users authentication by domain is described here.

Hi Max,
We will check on issue 1 regarding connect and disconnect too fast.
For item 2, based on our backend server log it received request from WCS after 25 seconds. Means its actually WCS is delaying sending to backend server.

This morning I encounter WCS hang by using custom app key and pointing to "http://localhost:8081/apps/EchoApp". It's back to normal after restarting service.

 

[Max]

based on our backend server log it received request from WCS after 25 seconds

Please, send us HTTP/HTTPS traffic dump betwen WCS and backend server, it may help to understand where delay is really from.

This morning I encounter WCS hang by using custom app key and pointing to "http://localhost:8081/apps/EchoApp". It's back to normal after restarting service.

Please, send us server log at moment when it happens.