Elasticsearch high CPU usage and stalling


#1

Hello,

I am facing an issue with Elasticsearch, I see how it goes till 176%
CPU usage in my server, so I was wondering whether this is a bug in this
release or in the stack itself. It does not happen inmediatelly but
after two hours I see these peaks on CPU usage that sometimes lead to an
unresponsive server due to the load.

Is someone facing up with this issue as well?

I have tried to configured elasticsearch to make use of less CPU and memory with these options

$# cat elasticsearch.yml 
indices.fieldata.cache.size: "30%"
indices.breaker.fielddata.limit: "80%"
indices.breaker.request.limit: "50%"
indices.breaker.total.limit: "90%"
#bootstrap.mlockall: true
index.merge.scheduler.max_thread_count: 1
indices.store.throttle.type: none

My java version is

$~# java -version
java version "1.8.0_101"
Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)

And a log I was recording with top to search for the CPU peaks

root@g:~# cat cpu-log.txt | grep colord

 **110  6949 colord   /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-2.3.5.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch start -Des.network.host=0.0.0.0**

**86.6  8100 colord   /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-2.3.5.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch start -Des.network.host=0.0.0.0**

#2

Finally, I have caught a trace of Elasticsearch, which is complaining of not enough memory for the JAva virtual machine

>     > /proc/meminfo:
>     > MemTotal:        2049916 kB
>     > MemFree:          136432 kB
>     > Buffers:          151092 kB
>     > Cached:            86336 kB
>     > SwapCached:            0 kB
>     > Active:          1417996 kB
>     > Inactive:          62320 kB
>     > Active(anon):    1246284 kB
>     > Inactive(anon):    20588 kB
>     > Active(file):     171712 kB
>     > Inactive(file):    41732 kB
>     > Unevictable:           0 kB
>     > Mlocked:               0 kB
>     > SwapTotal:             0 kB
>     > SwapFree:              0 kB
>     > Dirty:             23748 kB
>     > Writeback:             0 kB
>     > AnonPages:       1243096 kB
>     > Mapped:            33900 kB
>     > Shmem:             23912 kB
>     > Slab:             385480 kB
>     > SReclaimable:     353332 kB
>     > SUnreclaim:        32148 kB
>     > KernelStack:        4976 kB
>     > PageTables:        11276 kB
>     > NFS_Unstable:          0 kB
>     > Bounce:                0 kB
>     > WritebackTmp:          0 kB
>     > CommitLimit:     1024956 kB
>     > Committed_AS:    4504916 kB
>     > VmallocTotal:   34359738367 kB
>     > VmallocUsed:       10916 kB
>     > VmallocChunk:   34359707292 kB
>     > HardwareCorrupted:     0 kB
>     > AnonHugePages:    270336 kB
>     > HugePages_Total:       0
>     > HugePages_Free:        0
>     > HugePages_Rsvd:        0
>     > HugePages_Surp:        0
>     > Hugepagesize:       2048 kB
>     > DirectMap4k:       53240 kB
>     > DirectMap2M:     2043904 kB
>     > DirectMap1G:           0 kB
>     > processor    : 1
>     > vendor_id    : GenuineIntel
>     > cpu family    : 6
>     > model        : 45
>     > model name    : Intel(R) Xeon(R) CPU E5-2630L 0 @ 2.00GHz
>     > bogomips    : 3999.99
>     > clflush size    : 64
>     > cache_alignment    : 64
>     > address sizes    : 40 bits physical, 48 bits virtual
>     > power management:
>     > Memory: 4k page, physical 2049916k(136432k free), swap 0k(0k free)
>     > vm_info: OpenJDK 64-Bit Server VM (25.102-b14) for linux-amd64 JRE (1.8.0_102-8u102-b14.1-1~bpo8+1-b14), built on Aug  8 2016 10:49:23 by "jmm" with gcc 4.9.2
>     > time: Fri Oct  7 04:57:45 2016
>     > elapsed time: 0 seconds (0d 0h 0m 0s)
>     > #
>     > # There is insufficient memory for the Java Runtime Environment to continue.
>     > # Native memory allocation (mmap) failed to map 874119168 bytes for committing reserved memory.
>     > # Can not save log file, dump to screen..
>     > #
>     > # There is insufficient memory for the Java Runtime Environment to continue.
>     > # Native memory allocation (mmap) failed to map 874119168 bytes for committing reserved memory.
>     > # Possible reasons:
>     > #   The system is out of physical RAM or swap space
>     > #   In 32 bit mode, the process size limit was hit
>     > # Possible solutions:
>     > #   Reduce memory load on the system
>     > #   Increase physical memory or swap space
>     > #   Check if swap backing store is full
>     > #   Use 64 bit Java on a 64 bit OS
>     > #   Decrease Java heap size (-Xmx/-Xms)
>     > #   Decrease number of Java threads
>     > #   Decrease Java thread stack sizes (-Xss)
>     > #   Set larger code cache with -XX:ReservedCodeCacheSize=
>     > # This output file may be truncated or incomplete.
>     > #
>     > #  Out of Memory Error (os_linux.cpp:2630), pid=1, tid=0x00007fdb939a6700
>     > #
>     > # JRE version:  (8.0_102-b14) (build )
>     > # Java VM: OpenJDK 64-Bit Server VM (25.102-b14 mixed mode linux-amd64 compressed oops)
>     > # Core dump written. Default location: /usr/share/elasticsearch/core or core.1
>     > #
>     > ---------------  T H R E A D  ---------------
>     > Current thread (0x00007fdb8c00b000):  JavaThread "Unknown thread" [_thread_in_vm, id=21, stack(0x00007fdb938a6000,0x00007fdb939a7000)]
>     > Stack: [0x00007fdb938a6000,0x00007fdb939a7000],  sp=0x00007fdb939a5520,  free space=1021k
>     > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)

(system) #3