Virtual memory is increased for every second - Out of Memory Error

Muthu Source

I am having a Linux system with 8GB RAM (7.78 GB real and 1024 MB swap) with 4 cores. I am running the spring boot application with below command

screen -dmS API-Server java -jar -verbose:gc
-Dspring.profiles.active=prods1 -Xms1024m -Xmx4096m -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/ssvm/heapdump -XX:OnOutOfMemoryError='jmap -histo -F %p' /home/ssvm/api/nt-api-v2-0.0.1-SNAPSHOT.jar

I am having an Apache on this server, All the API request will be received by Apache and Apache will redirect to Java API Server. 2000 users will be using this API at the peek hours. Maximum RAM usage is 1.5 to 2.5 GB. But Virtual memory is increasing for every second (Monitored it on the TOP). At one point of time, I got Out of Memory Error for new api requests. CPU usage was 50% at peek time and RAM usage was : Apache - 1.36 GB and Java - 1.60 GB

I took the heapdump using jmap, I analyzed it using VisualVM, RAM usage was 1443 MB. I have Free RAM and CPU usage. I didn't see any issues on the code level. The only difference that I am seeing is, Virtual memory is increasing for every second, It reaches 10 GB, 20 GB, 26GB.

Any thoughts on this issue ? 1) Why the Virtual memory is increasing continuously ? 2) What could be the reason for this Out of Memory Error ?

[email protected]:~# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31861
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 350000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 31861
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited


[email protected]:~# java -XX:+PrintFlagsFinal -version | grep ThreadStackSize
     intx CompilerThreadStackSize                   = 0                                   {pd product}
     intx ThreadStackSize                           = 1024                                {pd product}
     intx VMThreadStackSize                         = 1024                                {pd product}
openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-0ubuntu0.16.04.1-b11)
OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode)


Error log is generated as below (Last line of the log clearly shows, It has sufficient memory):

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (gcTaskThread.cpp:48), pid=14493, tid=0x00007fefe3a1e700
#
# JRE version:  (8.0_171-b11) (build )
# Java VM: OpenJDK 64-Bit Server VM (25.171-b11 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#

---------------  T H R E A D  ---------------

Current thread (0x0000000000c07800):  JavaThread "Unknown thread" [_thread_in_vm, id=14493, stack(0x00007ffc0363f000,0x00007ffc0373f000)]

Stack: [0x00007ffc0363f000,0x00007ffc0373f000],  sp=0x00007ffc03739620,  free space=1001k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xa6db9b]
V  [libjvm.so+0x4c694c]
V  [libjvm.so+0x5a87ff]
V  [libjvm.so+0x5a7967]
V  [libjvm.so+0x8e0169]
V  [libjvm.so+0xa310f9]
V  [libjvm.so+0xa31342]
V  [libjvm.so+0x61133f]
V  [libjvm.so+0xa166ea]
V  [libjvm.so+0x68b8c2]  JNI_CreateJavaVM+0x62
C  [libjli.so+0x2f2e]
C  [libjli.so+0x84f6]
C  [libjli.so+0x4a44]
C  [libjli.so+0x55a0]  JLI_Launch+0x9d0
C  [java+0x621]
C  [libc.so.6+0x20830]  __libc_start_main+0xf0


---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )

Other Threads:

=>0x0000000000c07800 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=14493, stack(0x00007ffc0363f000,0x00007ffc0373f000)]

VM state:not at safepoint (not fully initialized)

VM Mutex/Monitor currently owned by a thread: None

GC Heap History (0 events):
No events

Deoptimization events (0 events):
No events

Classes redefined (0 events):
No events

Internal exceptions (0 events):
No events

Events (0 events):
No events


Dynamic libraries:
00400000-00401000 r-xp 00000000 08:00 256826                             /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
00600000-00601000 r--p 00000000 08:00 256826                             /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
00601000-00602000 rw-p 00001000 08:00 256826                             /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
00be8000-00c28000 rw-p 00000000 00:00 0                                  [heap]
83600000-88a00000 rw-p 00000000 00:00 0 
88a00000-d6780000 ---p 00000000 00:00 0 
d6780000-d9180000 rw-p 00000000 00:00 0 
d9180000-100000000 ---p 00000000 00:00 0 
7fefd093d000-7fefd0967000 rw-p 00000000 00:00 0 
7fefd0967000-7fefd0bd6000 ---p 00000000 00:00 0 
7fefd0bd6000-7fefd0c00000 rw-p 00000000 00:00 0 
7fefd0c00000-7fefd0e6e000 ---p 00000000 00:00 0 
7fefd0e6e000-7fefd0e84000 rw-p 00000000 00:00 0 
7fefd0e84000-7fefd0fbb000 ---p 00000000 00:00 0 
7fefd0fbb000-7fefd0fc6000 rw-p 00000000 00:00 0 
7fefd0fc6000-7fefd137c000 ---p 00000000 00:00 0 
7fefd137c000-7fefd15ec000 rwxp 00000000 00:00 0 
7fefd15ec000-7fefe037c000 ---p 00000000 00:00 0 
7fefe037c000-7fefe0384000 r-xp 00000000 08:00 256865                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/libzip.so
7fefe0384000-7fefe0583000 ---p 00008000 08:00 256865                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/libzip.so
7fefe0583000-7fefe0584000 r--p 00007000 08:00 256865                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/libzip.so
7fefe0584000-7fefe0585000 rw-p 00008000 08:00 256865                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/libzip.so
7fefe0585000-7fefe0590000 r-xp 00000000 08:00 4286                       /lib/x86_64-linux-gnu/libnss_files-2.23.so
7fefe0590000-7fefe078f000 ---p 0000b000 08:00 4286                       /lib/x86_64-linux-gnu/libnss_files-2.23.so
7fefe078f000-7fefe0790000 r--p 0000a000 08:00 4286                       /lib/x86_64-linux-gnu/libnss_files-2.23.so
7fefe0790000-7fefe0791000 rw-p 0000b000 08:00 4286                       /lib/x86_64-linux-gnu/libnss_files-2.23.so
7fefe0791000-7fefe0797000 rw-p 00000000 00:00 0 
7fefe0797000-7fefe07a2000 r-xp 00000000 08:00 4290                       /lib/x86_64-linux-gnu/libnss_nis-2.23.so
7fefe07a2000-7fefe09a1000 ---p 0000b000 08:00 4290                       /lib/x86_64-linux-gnu/libnss_nis-2.23.so
7fefe09a1000-7fefe09a2000 r--p 0000a000 08:00 4290                       /lib/x86_64-linux-gnu/libnss_nis-2.23.so
7fefe09a2000-7fefe09a3000 rw-p 0000b000 08:00 4290                       /lib/x86_64-linux-gnu/libnss_nis-2.23.so
7fefe09a3000-7fefe09b9000 r-xp 00000000 08:00 4264                       /lib/x86_64-linux-gnu/libnsl-2.23.so
7fefe09b9000-7fefe0bb8000 ---p 00016000 08:00 4264                       /lib/x86_64-linux-gnu/libnsl-2.23.so
7fefe0bb8000-7fefe0bb9000 r--p 00015000 08:00 4264                       /lib/x86_64-linux-gnu/libnsl-2.23.so
7fefe0bb9000-7fefe0bba000 rw-p 00016000 08:00 4264                       /lib/x86_64-linux-gnu/libnsl-2.23.so
7fefe0bba000-7fefe0bbc000 rw-p 00000000 00:00 0 
7fefe0bbc000-7fefe0bc4000 r-xp 00000000 08:00 6115                       /lib/x86_64-linux-gnu/libnss_compat-2.23.so
7fefe0bc4000-7fefe0dc3000 ---p 00008000 08:00 6115                       /lib/x86_64-linux-gnu/libnss_compat-2.23.so
7fefe0dc3000-7fefe0dc4000 r--p 00007000 08:00 6115                       /lib/x86_64-linux-gnu/libnss_compat-2.23.so
7fefe0dc4000-7fefe0dc5000 rw-p 00008000 08:00 6115                       /lib/x86_64-linux-gnu/libnss_compat-2.23.so
7fefe0dc5000-7fefe0df2000 r-xp 00000000 08:00 256856                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/libjava.so
7fefe0df2000-7fefe0ff1000 ---p 0002d000 08:00 256856                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/libjava.so
7fefe0ff1000-7fefe0ff2000 r--p 0002c000 08:00 256856                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/libjava.so
7fefe0ff2000-7fefe0ff4000 rw-p 0002d000 08:00 256856                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/libjava.so
7fefe0ff4000-7fefe1003000 r-xp 00000000 08:00 256842                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/libverify.so
7fefe1003000-7fefe1202000 ---p 0000f000 08:00 256842                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/libverify.so
7fefe1202000-7fefe1204000 r--p 0000e000 08:00 256842                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/libverify.so
7fefe1204000-7fefe1205000 rw-p 00010000 08:00 256842                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/libverify.so
7fefe1205000-7fefe120c000 r-xp 00000000 08:00 4292                       /lib/x86_64-linux-gnu/librt-2.23.so
7fefe120c000-7fefe140b000 ---p 00007000 08:00 4292                       /lib/x86_64-linux-gnu/librt-2.23.so
7fefe140b000-7fefe140c000 r--p 00006000 08:00 4292                       /lib/x86_64-linux-gnu/librt-2.23.so
7fefe140c000-7fefe140d000 rw-p 00007000 08:00 4292                       /lib/x86_64-linux-gnu/librt-2.23.so
7fefe140d000-7fefe1423000 r-xp 00000000 08:00 2256                       /lib/x86_64-linux-gnu/libgcc_s.so.1
7fefe1423000-7fefe1622000 ---p 00016000 08:00 2256                       /lib/x86_64-linux-gnu/libgcc_s.so.1
7fefe1622000-7fefe1623000 rw-p 00015000 08:00 2256                       /lib/x86_64-linux-gnu/libgcc_s.so.1
7fefe1623000-7fefe172b000 r-xp 00000000 08:00 6553                       /lib/x86_64-linux-gnu/libm-2.23.so
7fefe172b000-7fefe192a000 ---p 00108000 08:00 6553                       /lib/x86_64-linux-gnu/libm-2.23.so
7fefe192a000-7fefe192b000 r--p 00107000 08:00 6553                       /lib/x86_64-linux-gnu/libm-2.23.so
7fefe192b000-7fefe192c000 rw-p 00108000 08:00 6553                       /lib/x86_64-linux-gnu/libm-2.23.so
7fefe192c000-7fefe1a9e000 r-xp 00000000 08:00 24061                      /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7fefe1a9e000-7fefe1c9e000 ---p 00172000 08:00 24061                      /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7fefe1c9e000-7fefe1ca8000 r--p 00172000 08:00 24061                      /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7fefe1ca8000-7fefe1caa000 rw-p 0017c000 08:00 24061                      /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7fefe1caa000-7fefe1cae000 rw-p 00000000 00:00 0 
7fefe1cae000-7fefe2907000 r-xp 00000000 08:00 256869                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
7fefe2907000-7fefe2b07000 ---p 00c59000 08:00 256869                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
7fefe2b07000-7fefe2b9a000 r--p 00c59000 08:00 256869                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
7fefe2b9a000-7fefe2bc3000 rw-p 00cec000 08:00 256869                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
7fefe2bc3000-7fefe2bf3000 rw-p 00000000 00:00 0 
7fefe2bf3000-7fefe2c0b000 r-xp 00000000 08:00 4289                       /lib/x86_64-linux-gnu/libpthread-2.23.so
7fefe2c0b000-7fefe2e0a000 ---p 00018000 08:00 4289                       /lib/x86_64-linux-gnu/libpthread-2.23.so
7fefe2e0a000-7fefe2e0b000 r--p 00017000 08:00 4289                       /lib/x86_64-linux-gnu/libpthread-2.23.so
7fefe2e0b000-7fefe2e0c000 rw-p 00018000 08:00 4289                       /lib/x86_64-linux-gnu/libpthread-2.23.so
7fefe2e0c000-7fefe2e10000 rw-p 00000000 00:00 0 
7fefe2e10000-7fefe2e13000 r-xp 00000000 08:00 6114                       /lib/x86_64-linux-gnu/libdl-2.23.so
7fefe2e13000-7fefe3012000 ---p 00003000 08:00 6114                       /lib/x86_64-linux-gnu/libdl-2.23.so
7fefe3012000-7fefe3013000 r--p 00002000 08:00 6114                       /lib/x86_64-linux-gnu/libdl-2.23.so
7fefe3013000-7fefe3014000 rw-p 00003000 08:00 6114                       /lib/x86_64-linux-gnu/libdl-2.23.so
7fefe3014000-7fefe302d000 r-xp 00000000 08:00 2375                       /lib/x86_64-linux-gnu/libz.so.1.2.8
7fefe302d000-7fefe322c000 ---p 00019000 08:00 2375                       /lib/x86_64-linux-gnu/libz.so.1.2.8
7fefe322c000-7fefe322d000 r--p 00018000 08:00 2375                       /lib/x86_64-linux-gnu/libz.so.1.2.8
7fefe322d000-7fefe322e000 rw-p 00019000 08:00 2375                       /lib/x86_64-linux-gnu/libz.so.1.2.8
7fefe322e000-7fefe33ee000 r-xp 00000000 08:00 6558                       /lib/x86_64-linux-gnu/libc-2.23.so
7fefe33ee000-7fefe35ee000 ---p 001c0000 08:00 6558                       /lib/x86_64-linux-gnu/libc-2.23.so
7fefe35ee000-7fefe35f2000 r--p 001c0000 08:00 6558                       /lib/x86_64-linux-gnu/libc-2.23.so
7fefe35f2000-7fefe35f4000 rw-p 001c4000 08:00 6558                       /lib/x86_64-linux-gnu/libc-2.23.so
7fefe35f4000-7fefe35f8000 rw-p 00000000 00:00 0 
7fefe35f8000-7fefe3605000 r-xp 00000000 08:00 256866                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/jli/libjli.so
7fefe3605000-7fefe3805000 ---p 0000d000 08:00 256866                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/jli/libjli.so
7fefe3805000-7fefe3806000 r--p 0000d000 08:00 256866                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/jli/libjli.so
7fefe3806000-7fefe3807000 rw-p 0000e000 08:00 256866                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/jli/libjli.so
7fefe3807000-7fefe382d000 r-xp 00000000 08:00 4267                       /lib/x86_64-linux-gnu/ld-2.23.so
7fefe3914000-7fefe391c000 rw-s 00000000 08:00 375248                     /tmp/hsperfdata_ssvm/14493
7fefe391c000-7fefe391d000 ---p 00000000 00:00 0 
7fefe391d000-7fefe3a22000 rw-p 00000000 00:00 0 
7fefe3a28000-7fefe3a29000 rw-p 00000000 00:00 0 
7fefe3a29000-7fefe3a2a000 r--p 00000000 00:00 0 
7fefe3a2a000-7fefe3a2c000 rw-p 00000000 00:00 0 
7fefe3a2c000-7fefe3a2d000 r--p 00025000 08:00 4267                       /lib/x86_64-linux-gnu/ld-2.23.so
7fefe3a2d000-7fefe3a2e000 rw-p 00026000 08:00 4267                       /lib/x86_64-linux-gnu/ld-2.23.so
7fefe3a2e000-7fefe3a2f000 rw-p 00000000 00:00 0 
7ffc0363f000-7ffc03642000 ---p 00000000 00:00 0 
7ffc03642000-7ffc0373f000 rw-p 00000000 00:00 0                          [stack]
7ffc0377f000-7ffc03782000 r--p 00000000 00:00 0                          [vvar]
7ffc03782000-7ffc03784000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

VM Arguments:
java_command: <unknown>
java_class_path (initial): .
Launcher Type: SUN_STANDARD

Environment Variables:
PATH=/home/ssvm/bin:/home/ssvm/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
SHELL=/bin/bash

Signal Handlers:
SIGSEGV: [libjvm.so+0xa6e6f0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGBUS: [libjvm.so+0xa6e6f0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGFPE: [libjvm.so+0x8b5d50], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGPIPE: [libjvm.so+0x8b5d50], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGXFSZ: [libjvm.so+0x8b5d50], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGILL: [libjvm.so+0x8b5d50], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGUSR1: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGUSR2: [libjvm.so+0x8b5c00], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
SIGHUP: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGINT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGTERM: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGQUIT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none


---------------  S Y S T E M  ---------------

OS:DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.2 LTS"

uname:Linux 4.15.13-x86_64-linode106 #1 SMP Tue Mar 27 14:42:14 UTC 2018 x86_64
libc:glibc 2.23 NPTL 2.23 
rlimit: STACK 8192k, CORE 0k, NPROC 30000, NOFILE 350000, AS infinity
load average:0.12 0.61 0.78

/proc/meminfo:
MemTotal:        8162148 kB
MemFree:          926104 kB
MemAvailable:    4199816 kB
Buffers:           36152 kB
Cached:          3239184 kB
SwapCached:            0 kB
Active:          3805844 kB
Inactive:        2607852 kB
Active(anon):    3146828 kB
Inactive(anon):    22700 kB
Active(file):     659016 kB
Inactive(file):  2585152 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:        262140 kB
SwapFree:         262140 kB
Dirty:               312 kB
Writeback:             0 kB
AnonPages:       3138376 kB
Mapped:            78500 kB
Shmem:             31172 kB
Slab:             258780 kB
SReclaimable:     136292 kB
SUnreclaim:       122488 kB
KernelStack:      207616 kB
PageTables:        69452 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     4343212 kB
Committed_AS:   14450168 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
DirectMap4k:       40824 kB
DirectMap2M:     5201920 kB
DirectMap1G:     5242880 kB


CPU:total 4 (initial active 4) (1 cores per cpu, 1 threads per core) family 6 model 63 stepping 2, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, lzcnt, tsc, bmi1, bmi2

/proc/cpuinfo:
processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model       : 63
model name  : Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
stepping    : 2
microcode   : 0x1
cpu MHz     : 2499.988
cache size  : 16384 KB
physical id : 0
siblings    : 1
core id     : 0
cpu cores   : 1
apicid      : 0
initial apicid  : 0
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat
bugs        : cpu_meltdown spectre_v1 spectre_v2
bogomips    : 5001.30
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 1
vendor_id   : GenuineIntel
cpu family  : 6
model       : 63
model name  : Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
stepping    : 2
microcode   : 0x1
cpu MHz     : 2499.988
cache size  : 16384 KB
physical id : 1
siblings    : 1
core id     : 0
cpu cores   : 1
apicid      : 1
initial apicid  : 1
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat
bugs        : cpu_meltdown spectre_v1 spectre_v2
bogomips    : 5001.30
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 2
vendor_id   : GenuineIntel
cpu family  : 6
model       : 63
model name  : Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
stepping    : 2
microcode   : 0x1
cpu MHz     : 2499.988
cache size  : 16384 KB
physical id : 2
siblings    : 1
core id     : 0
cpu cores   : 1
apicid      : 2
initial apicid  : 2
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat
bugs        : cpu_meltdown spectre_v1 spectre_v2
bogomips    : 5001.30
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 3
vendor_id   : GenuineIntel
cpu family  : 6
model       : 63
model name  : Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
stepping    : 2
microcode   : 0x1
cpu MHz     : 2499.988
cache size  : 16384 KB
physical id : 3
siblings    : 1
core id     : 0
cpu cores   : 1
apicid      : 3
initial apicid  : 3
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat
bugs        : cpu_meltdown spectre_v1 spectre_v2
bogomips    : 5001.30
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:



Memory: 4k page, physical 8162148k(926104k free), swap 262140k(262140k free)

vm_info: OpenJDK 64-Bit Server VM (25.171-b11) for linux-amd64 JRE (1.8.0_171-8u171-b11-0ubuntu0.16.04.1-b11), built on Apr 27 2018 17:19:03 by "buildd" with gcc 5.4.0 20160609

time: Wed Jun 13 11:33:33 2018
elapsed time: 0 seconds (0d 0h 0m 0s)
javalinuxjvmheap-memoryvirtual-memory

Answers

answered 4 months ago Muthu #1

Application is using a third party library which created threads and didn't destroy threads after processing is done. This created huge living threads. So the Virtual memory is increased and system crashed.

Found it using the jstack.

We can use the below command to check the running threads in a linux machine, ps -elfT | wc -l

it was increasing gradually.

When I analyse the jstack report found that huge threads were created and not destroyed properly.

comments powered by Disqus