Problem accessing web service

burak guder

burak guder

Member
Hİ

I have a 32 core server with an average of 400 people connecting to
total bitrate 350kbit

Although there was no problem on the stream side, the web service crashed and the embet player did not open.

It worked when I restarted the webcallserver again. I see the following error numbers on the server logs.



11:02:37,224 ERROR HttpServerHandler - HTTPS-pool-5-thread-768 HTTP error
javax.net.ssl.SSLException: Session has no PSK
at sun.security.ssl.Alert.createSSLException(Alert.java:133)
at sun.security.ssl.Alert.createSSLException(Alert.java:117)
at sun.security.ssl.TransportContext.fatal(TransportContext.java:311)
at sun.security.ssl.TransportContext.fatal(TransportContext.java:267)
at sun.security.ssl.TransportContext.fatal(TransportContext.java:258)
at sun.security.ssl.PreSharedKeyExtension.checkBinder(PreSharedKeyExtension.java:535)
at sun.security.ssl.PreSharedKeyExtension.access$1000(PreSharedKeyExtension.java:50)
at sun.security.ssl.PreSharedKeyExtension$CHPreSharedKeyUpdate.consume(PreSharedKeyExtension.java:526)
at sun.security.ssl.SSLExtension.consumeOnTrade(SSLExtension.java:562)
at sun.security.ssl.SSLExtensions.consumeOnTrade(SSLExtensions.java:222)
at sun.security.ssl.ServerHello$T13ServerHelloProducer.produce(ServerHello.java:540)
at sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:421)
at sun.security.ssl.ClientHello$T13ClientHelloConsumer.goServerHello(ClientHello.java:1152)
at sun.security.ssl.ClientHello$T13ClientHelloConsumer.consume(ClientHello.java:1088)
at sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:725)
at sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:693)
at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377)
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:968)
at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:902)
at org.jboss.netty.handler.ssl.SslHandler$2.run(Unknown Source)
at org.jboss.netty.handler.ssl.ImmediateExecutor.execute(Unknown Source)
at org.jboss.netty.handler.ssl.SslHandler.runDelegatedTasks(Unknown Source)
at org.jboss.netty.handler.ssl.SslHandler.unwrap(Unknown Source)
at org.jboss.netty.handler.ssl.SslHandler.decode(Unknown Source)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(Unknown Source)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(Unknown Source)
at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
at org.jboss.netty.channel.socket.nio.NioWorker.read(Unknown Source)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(Unknown Source)
at org.jboss.netty.channel.socket.nio.DeadlockAwareNioWorker.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

____________________________________________________________________________


11:09:12,293 ERROR HttpServerHandler - HTTPS-pool-5-thread-1125 HTTP error
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:695)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:241)
at sun.nio.ch.IOUtil.read(IOUtil.java:195)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:159)
at com.flashphoner.server.http.J.A(Unknown Source)
at com.flashphoner.server.http.H.A(Unknown Source)
at com.flashphoner.server.http.handlers.SharedRequestHandler.process(Unknown Source)
at com.flashphoner.server.http.G.messageReceived(Unknown Source)
at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(Unknown Source)
at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(Unknown Source)
at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(Unknown Source)
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(Unknown Source)
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(Unknown Source)
at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(Unknown Source)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(Unknown Source)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(Unknown Source)
at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
at org.jboss.netty.channel.Channels.fireMessageReceived(Unknown Source)
at org.jboss.netty.channel.socket.nio.NioWorker.read(Unknown Source)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(Unknown Source)
at org.jboss.netty.channel.socket.nio.DeadlockAwareNioWorker.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

__________________________________________________________________________________________

11:44:43,011 ERROR HttpServerHandler - HTTPS-pool-5-thread-152 HTTP error
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:51)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:470)
at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$PooledSendBuffer.transferTo(Unknown Source)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(Unknown Source)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromSelectorLoop(Unknown Source)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(Unknown Source)
at org.jboss.netty.channel.socket.nio.DeadlockAwareNioWorker.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)


flashphoner config

# Config flashphoner.properties
# To get more settings:
# ssh -p 2001 admin@localhost
# default password: admin
# show node-settings
# show node-settings | grep port

#server ip
ip =xxxxxx
ip_local =xxxxxx

#webrtc ports range
media_port_from =55540
media_port_to =63000

rtmp_port_from =63001
rtmp_port_to =63100

rtsp_port_from =63101
rtsp_port_to =63600

wcs_agent_port_from =63601
wcs_agent_port_to =64000

port_from =64001
port_to =65535


#codecs
codecs =opus,alaw,ulaw,g729,speex16,g722,mpeg4-generic,telephone-event,h264,vp8,flv,mpv
codecs_exclude_sip =mpeg4-generic,flv,mpv
codecs_exclude_streaming =flv,telephone-event
codecs_exclude_sip_rtmp =opus,g729,g722,mpeg4-generic,vp8,mpv

#websocket ports
ws.port =9080
wss.port =9443

#rtmp port
rtmfp.port =1936
rtmp.port =1936

hls.http.port =9082
hls.https.port =9445

rtp_force_synchronization=true
disable_manager_rmi=false
#disable_rest_auth=false
disable_drop_aac_frame=true


rest_access_control_allow_origin=*
rest_access_control_allow_headers=content-type,x-requested-with
rest_access_control_allow_methods=POST

rtp_receive_buffer_size=131072
rtp_send_buffer_size =131072

rtsp_activity_timer_timeout=86400000

rest_max_connections=3000
disable_rest_requests=true

http.port=9082
https.port=9444

______________________________________________-

system features

openjdk version "1.8.0_292"
OpenJDK Runtime Environment (build 1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10)
OpenJDK 64-Bit Server VM (build 25.292-b10, mixed mode)


UBUNTU 16.04

CPU Use % 503
RAM Use %1.6

Run Code

flashph+ 43540 471 1.6 5377296 2112016 ? Sl 11:34 182:43 java -Xmx1024M -XX:+UseConcMarkSweepGC -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=50999 -Dcom.sun.management.jmxremote.host=localhost -Djava.rmi.server.hostname=localhost -XX:ErrorFile=/usr/local/FlashphonerWebCallServer/logs/error%p.log -XX:+PrintGCDateStamps -XX:+PrintGCDetails -Xloggc:/usr/local/FlashphonerWebCallServer/logs/gc-core-2021-06-02_11-34.log -XX:+ExplicitGCInvokesConcurrent -Dsun.rmi.dgc.client.gcInterval=36000000000 -Dsun.rmi.dgc.server.gcInterval=36000000000 -Dcom.flashphoner.fms.AppHome=/usr/local/FlashphonerWebCallServer -Djava.library.path=/usr/local/FlashphonerWebCallServer/lib/so:/usr/local/FlashphonerWebCallServer/lib -DWCS_NON_ROOT=true -DsessionDebugEnabled=false -Djdk.tls.client.protocols="TLSv1,TLSv1.1,TLSv1.2" -cp /usr/local/FlashphonerWebCallServer/lib/* com.flashphoner.server.Server

NOTE
Application CPU usage does not exceed 500%. Is there a way to increase the number of threads?
 
Last edited:
Max

Max

Administrator
Staff member
Good day.
Please clarify how much memory on your server, and how much memory is allocated for Java heap.
Ubuntu 186.04 is not officially tested with 5.2 branch, so we strongly recommend to upgrade to Ubuntu 18.04. See all the officially supported Linux distrubution packages list here
Also, to work under load, we recommend to update Java to 12 or 14 and set up ZGC garbage collector as described here. You probably shuold also tune physical memory allocation as described here.
Try also the following settings:
Code: 
ice_tcp_nio=false
ice_udp_nio=false
 
burak guder

burak guder

Member
Hi


post update code is below

java -Xmx32g -XX:+UnlockExperimentalVMOptions -XX:+UseZGC -Xms64g -Xmx64g -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=50999 -Dcom.sun.management.jmxremote.host=localhost -Djava.rmi.server.hostname=localhost -XX:ErrorFile=/usr/local/FlashphonerWebCallServer/logs/error%p.log -Xlog:gc*:/usr/local/FlashphonerWebCallServer/logs/gc-core-2021-06-02_23-38.log:time -XX:+ExplicitGCInvokesConcurrent -Dsun.rmi.dgc.client.gcInterval=36000000000 -Dsun.rmi.dgc.server.gcInterval=36000000000 -Dcom.flashphoner.fms.AppHome=/usr/local/FlashphonerWebCallServer -Djava.library.path=/usr/local/FlashphonerWebCallServer/lib/so:/usr/local/FlashphonerWebCallServer/lib -DWCS_NON_ROOT=true -DsessionDebugEnabled=false -Djdk.tls.client.protocols="TLSv1,TLSv1.1,TLSv1.2" -cp /usr/local/FlashphonerWebCallServer/lib/* com.flashphoner.server.Server

--- OS upgraded to ubuntu 18.04

--- java --version
openjdk 12.0.2 2019-07-16
OpenJDK Runtime Environment (build 12.0.2+10)
OpenJDK 64-Bit Server VM (build 12.0.2+10, mixed mode, sharing)


Server :
32 Core 32 thread
RAM : 128 GB
1 Gbit Port

Are the values entered for the server normal? Or suggest another value?
 
burak guder

burak guder

Member
I deleted the initial -Xmx32g statement.



java -XX:+UnlockExperimentalVMOptions -XX:+UseZGC -Xms64g -Xmx64g -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=50999 -Dcom.sun.management.jmxremote.host=localhost -Djava.rmi.server.hostname=localhost -XX:ErrorFile=/usr/local/FlashphonerWebCallServer/logs/error%p.log -Xlog:gc*:/usr/local/FlashphonerWebCallServer/logs/gc-core-2021-06-03_08-47.log:time -XX:+ExplicitGCInvokesConcurrent -Dsun.rmi.dgc.client.gcInterval=36000000000 -Dsun.rmi.dgc.server.gcInterval=36000000000 -Dcom.flashphoner.fms.AppHome=/usr/local/FlashphonerWebCallServer -Djava.library.path=/usr/local/FlashphonerWebCallServer/lib/so:/usr/local/FlashphonerWebCallServer/lib -DWCS_NON_ROOT=true -DsessionDebugEnabled=false -Djdk.tls.client.protocols="TLSv1,TLSv1.1,TLSv1.2" -cp /usr/local/FlashphonerWebCallServer/lib/* com.flashphoner.server.Server

today it will work again under high load. i write the results

Thanks for be interested
 
You must log in or register to reply here.
Top