WCS stop working every 2 weeks (more or less)

mbedial

Member
Hello,
we have a voice service that we developed for a client based on WCS (we have the commercial license). According to our CDRs we have 300-400 minutes everyday, but during the last 2 months , we've had to restart the WCS more or less every 15 days , since it stopped working.
The last restart was yesterday although it stopped the day before (more or less the last call wast June the 17th at 14:39 spanish time.
In the logs at that time I can see all the time:

3:01:37,245 WARN WSServerHandler - WSS-pool-8-thread-42 Close channel [id: 0x44fd7b24, /93.43.x.x:10615 :> /90.169.x.x:9443] because: java.lang.OutOfMemoryError 'Java heap space'
13:17:27,282 INFO WSServerHandler - WSS-pool-8-thread-42 Channel already closed, do nothing
12:59:17,601 WARN adlockAwareNioWorker - WSS-pool-8-thread-67 Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: Java heap space
12:59:10,391 WARN adlockAwareNioWorker - WSS-pool-8-thread-50 Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: Java heap space
12:59:04,041 INFO SipUserAgentListener - pool-22-thread-8 Thread is out of order. Waiting for correction. Seq = 800385
12:59:04,041 WARN adlockAwareNioWorker - WSS-pool-8-thread-61 Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: Java heap space
12:58:46,063 WARN adlockAwareNioWorker - WS-pool-6-thread-4 Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: Java heap space
12:58:35,573 WARN adlockAwareNioWorker - WSS-pool-8-thread-31 Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: Java heap space
12:57:43,974 INFO SipUserAgentListener - pool-22-thread-1 Thread is out of order. Waiting for correction. Seq = 800378
12:52:08,363 WARN WSServerHandler - WSS-pool-8-thread-71 Close channel [id: 0x35db6fb9, /37.171.x.x:7723 => /90.169.x.x:9443] because: java.lang.OutOfMemoryError 'Java heap space'
12:51:21,420 INFO SipUserAgentListener - pool-22-thread-3 Thread is out of order. Waiting for correction. Seq = 800377
12:48:01,224 INFO SipUserAgentListener - pool-22-thread-10 Thread is out of order. Waiting for correction. Seq = 800375
12:46:21,687 INFO SipUserAgentListener - pool-22-thread-2 Thread is out of order. Waiting for correction. Seq = 800382
12:44:49,100 ERROR C - Flashphoner-AbstractRtpSession-Timer Caught exception on channel [id: 0x6e694b1f, /10.0.0.4:41227].
java.net.SocketException: Socket is closed
at java.net.DatagramSocket.send(DatagramSocket.java:658)
at com.flashphoner.C.C.B.D.A(Unknown Source)
at com.flashphoner.C.C.B.E.eventSunk(Unknown Source)
at org.jboss.netty.channel.Channels.write(Unknown Source)
at com.flashphoner.C.D.B.D.handleDownstream(Unknown Source)
at org.jboss.netty.channel.Channels.write(Unknown Source)
at org.jboss.netty.channel.AbstractChannel.write(Unknown Source)
at com.flashphoner.C.C.B.I.write(Unknown Source)
at com.flashphoner.C.E.B.A(Unknown Source)
at com.flashphoner.C.E.B$5.A(Unknown Source)
at com.flashphoner.C.A.A.A(Unknown Source)
at com.flashphoner.C.E.B.B(Unknown Source)
at com.flashphoner.C.E.B$2.A(Unknown Source)
at com.flashphoner.C.A.A.A(Unknown Source)
at com.flashphoner.C.E.B.run(Unknown Source)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
 

Max

Administrator
Staff member
Hello
1. What is your WCS server version?
2. How much physical RAM does server have?
3. Please share WCS_HOME/conf/wcs-core.properties
4. Try to setup heap memory in wcs-core.properties
Code:
-Xmx8g -Xms8g
Here 8g should be a half of your physical RAM (i.e. 16 gb total RAM).
If you have older WCS version, try download latest available WCS version and check tuning settings in wcs-core.properties
https://flashphoner.com/download
 

mbedial

Member
Hi Max,
as far as I know we have 5.0.2570.
16 GB RAM:
less /proc/meminfo
MemTotal: 16392340 kB
MemFree: 537892 kB
Buffers: 154660 kB
Cached: 13787012 kB
SwapCached: 16112 kB
Active: 7660696 kB
Inactive: 7391980 kB
Active(anon): 490976 kB
Inactive(anon): 632604 kB
Active(file): 7169720 kB
Inactive(file): 6759376 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 16744444 kB
SwapFree: 16422188 kB
Dirty: 1876 kB
Writeback: 0 kB
AnonPages: 1097888 kB
Mapped: 49352 kB
Shmem: 12572 kB
Slab: 590692 kB
SReclaimable: 540980 kB
SUnreclaim: 49712 kB

Additional info:
:/usr/local$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 62
Stepping: 4
CPU MHz: 1200.000
BogoMIPS: 4788.33
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 10240K
NUMA node0 CPU(s): 0-3

See the atached file with the wsc-core.conf

BR
Mario
 

Attachments

Max

Administrator
Staff member
Try to configure in wcs-core.properties
Code:
-Xmx8g -Xms8g
instead of
Code:
-Xmx1024M
 
Top