Hello,
I am facing an issue with Elasticsearch, I see how it goes till 176%
CPU usage in my server, so I was wondering whether this is a bug in this
release or in the stack itself. It does not happen inmediatelly but
after two hours I see these peaks on CPU usage that sometimes lead to an
unresponsive server due to the load.
Is someone facing up with this issue as well?
I have tried to configured elasticsearch to make use of less CPU and memory with these options
$# cat elasticsearch.yml
indices.fieldata.cache.size: "30%"
indices.breaker.fielddata.limit: "80%"
indices.breaker.request.limit: "50%"
indices.breaker.total.limit: "90%"
#bootstrap.mlockall: true
index.merge.scheduler.max_thread_count: 1
indices.store.throttle.type: none
My java version is
$~# java -version
java version "1.8.0_101"
Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)
And a log I was recording with top to search for the CPU peaks
root@g:~# cat cpu-log.txt | grep colord
**110 6949 colord /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-2.3.5.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch start -Des.network.host=0.0.0.0**
**86.6 8100 colord /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-2.3.5.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch start -Des.network.host=0.0.0.0**
1 Like
Finally, I have caught a trace of Elasticsearch, which is complaining of not enough memory for the JAva virtual machine
> > /proc/meminfo:
> > MemTotal: 2049916 kB
> > MemFree: 136432 kB
> > Buffers: 151092 kB
> > Cached: 86336 kB
> > SwapCached: 0 kB
> > Active: 1417996 kB
> > Inactive: 62320 kB
> > Active(anon): 1246284 kB
> > Inactive(anon): 20588 kB
> > Active(file): 171712 kB
> > Inactive(file): 41732 kB
> > Unevictable: 0 kB
> > Mlocked: 0 kB
> > SwapTotal: 0 kB
> > SwapFree: 0 kB
> > Dirty: 23748 kB
> > Writeback: 0 kB
> > AnonPages: 1243096 kB
> > Mapped: 33900 kB
> > Shmem: 23912 kB
> > Slab: 385480 kB
> > SReclaimable: 353332 kB
> > SUnreclaim: 32148 kB
> > KernelStack: 4976 kB
> > PageTables: 11276 kB
> > NFS_Unstable: 0 kB
> > Bounce: 0 kB
> > WritebackTmp: 0 kB
> > CommitLimit: 1024956 kB
> > Committed_AS: 4504916 kB
> > VmallocTotal: 34359738367 kB
> > VmallocUsed: 10916 kB
> > VmallocChunk: 34359707292 kB
> > HardwareCorrupted: 0 kB
> > AnonHugePages: 270336 kB
> > HugePages_Total: 0
> > HugePages_Free: 0
> > HugePages_Rsvd: 0
> > HugePages_Surp: 0
> > Hugepagesize: 2048 kB
> > DirectMap4k: 53240 kB
> > DirectMap2M: 2043904 kB
> > DirectMap1G: 0 kB
> > processor : 1
> > vendor_id : GenuineIntel
> > cpu family : 6
> > model : 45
> > model name : Intel(R) Xeon(R) CPU E5-2630L 0 @ 2.00GHz
> > bogomips : 3999.99
> > clflush size : 64
> > cache_alignment : 64
> > address sizes : 40 bits physical, 48 bits virtual
> > power management:
> > Memory: 4k page, physical 2049916k(136432k free), swap 0k(0k free)
> > vm_info: OpenJDK 64-Bit Server VM (25.102-b14) for linux-amd64 JRE (1.8.0_102-8u102-b14.1-1~bpo8+1-b14), built on Aug 8 2016 10:49:23 by "jmm" with gcc 4.9.2
> > time: Fri Oct 7 04:57:45 2016
> > elapsed time: 0 seconds (0d 0h 0m 0s)
> > #
> > # There is insufficient memory for the Java Runtime Environment to continue.
> > # Native memory allocation (mmap) failed to map 874119168 bytes for committing reserved memory.
> > # Can not save log file, dump to screen..
> > #
> > # There is insufficient memory for the Java Runtime Environment to continue.
> > # Native memory allocation (mmap) failed to map 874119168 bytes for committing reserved memory.
> > # Possible reasons:
> > # The system is out of physical RAM or swap space
> > # In 32 bit mode, the process size limit was hit
> > # Possible solutions:
> > # Reduce memory load on the system
> > # Increase physical memory or swap space
> > # Check if swap backing store is full
> > # Use 64 bit Java on a 64 bit OS
> > # Decrease Java heap size (-Xmx/-Xms)
> > # Decrease number of Java threads
> > # Decrease Java thread stack sizes (-Xss)
> > # Set larger code cache with -XX:ReservedCodeCacheSize=
> > # This output file may be truncated or incomplete.
> > #
> > # Out of Memory Error (os_linux.cpp:2630), pid=1, tid=0x00007fdb939a6700
> > #
> > # JRE version: (8.0_102-b14) (build )
> > # Java VM: OpenJDK 64-Bit Server VM (25.102-b14 mixed mode linux-amd64 compressed oops)
> > # Core dump written. Default location: /usr/share/elasticsearch/core or core.1
> > #
> > --------------- T H R E A D ---------------
> > Current thread (0x00007fdb8c00b000): JavaThread "Unknown thread" [_thread_in_vm, id=21, stack(0x00007fdb938a6000,0x00007fdb939a7000)]
> > Stack: [0x00007fdb938a6000,0x00007fdb939a7000], sp=0x00007fdb939a5520, free space=1021k
> > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
1 Like