High CPU usages

Elastic Stack Elasticsearch
1

I have elastisearch installed on my server size of the which is 10GB
roughly, I have 16GB of RAM and 1TB of space. But still there are couple
of strange issues that I am unable to understand about.

  1. I have dedicated 10GB of ram to elasticsearch still my shards goes
    down and shows out of memory error when there is heavy insert
    operation

  2. happening.Randomly elasticsearch stats using maximum resources of my
    server as shown shown below as a result server performance is
    almost dead

    Screen Shot 2015-06-22 at 11.04.04 pm.png730×378 355 KB

  3. I am seeing this when i do ps -aef | grep elasticsearch
    HeapDumpOnOutOfMemoryError

    498 2838 8.3 4.7 17978352 781976 ? Sl 20:21 0:16 /usr/bin/java -Xms10G -Xmx10G -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Delasticsearch -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.5.1.jar:/usr/share/elasticsearch/lib/:/usr/share/elasticsearch/lib/sigar/ -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
    root 3258 0.0 0.0 103212 792 pts/0 S+ 20:24 0:00 grep elasticsearch

2

Unfortunately you cut off the load average, but I see your dilemma. Is ES and these perl apps running on the same server? you probably want to split them apart as they appear for contenting for the same resource. (CPU). Since there is no IOWAIT this looks to be completely a CPU issue.

As the JAVA heap fills up it has to spend more time to clear memory, which takes CPU So as it gets overloaded its going to slow everything down. Unfortunately.

So first, lets look at the details of ES, How many Nodes do you have? Have you configured separate master and data nodes?

Second, what do you consider heavy indexing? 1k, 1m, 1 billion? requests. ? What is your average indexing rate?
You may want to look at this using "head", Bigdesk, Marvel, or kopf, to get a better grasp of what is going on internally.
https://www.elastic.co/guide/en/elasticsearch/client/community/current/health.html

3

I have no perl apps running on the server. the perl scripts are are some ES scripts. Second I am new to ES and I have just gone with the basic installation of ES so I don't have any idea about how to configure separate master and data nodes. What should I do so JAVA heap should take less CPU while clearing. I even tried clearing cash with the api but dosen't help.

My index size would be approx 10m, And I don't know how to find out index rate and request

Here is my cluster details if may help

 {    "cluster_name": "elasticsearch",   
 "status": "yellow",   
 "timed_out": false,
    "number_of_nodes": 1,    
"number_of_data_nodes": 1,   
 "active_primary_shards": 54,  
  "active_shards": 54,  
  "relocating_shards": 0,   
 "initializing_shards": 0,
    "unassigned_shards": 56,   
 "number_of_pending_tasks": 0}
4

Elasticsearch does not use perl scripts. To see what these scripts are can you runthe following on your server?

ps auwx | grep perl

Some other questions for you:

  • Have you made any changes to your elasticsearch.yml file? such as enabling dynamic scripting? (If you don't know, maybe you could paste its contents into a gist, removing any sensitive info like IP addresses, and put a link to it here?)
  • Is you the box you are running Elasticsearch on accessible via the internet?
5

  1. Have you made any changes to your elasticsearch.yml file? such as
    enabling dynamic scripting? (If you don't know, maybe you could
    paste its contents into a gist, removing any sensitive info like IP
    addresses, and put a link to it here?)
    -Yes enabled dynamic scripting in order my ctx._source.Added+=1 query to work is it causing the problem, And what is solution for it ?

  2. Is you the box you are running Elasticsearch on accessible via the
    internet?

-Yes

But he above screenshot shows all the precess are of elastic search causing high CPU usages with perl command

6

The screenshot you pasted shows that all the perl processes are being run as the 'elasticsearch' user but I can assure you that Elasticsearch itself does not run any perl scripts.

By running Elasticsearch open on the internet, with default settings and enabling dynamic scripting you have exposed yourself to security vunerabilities. These perl scripts are probably being run using the dynamic scripting exploits. Did you run the command I posted above to see what the scripts being run are?

You should carefully read an implement all the suggestions on this blog post. Elasticsearch should not be run directly exposed to the internet, you should run it behind a proxy, firewall or similar security method. You should also change your cluster name from the default 'elasticsearch'. The blog post is fairly old so you should also read the scripting documentation for the current information regarding dynamic scripting. Note that you can disable dynamic scripting and run file scripts.

Finally, all the known security issues for Elasticsearch can be found here (and also mentioned in release notes where relevant), together with their CVE numbers and links to the official CVE issue. Dynamic scripting is disabled by default because of this CVE security issue.

7

Still I am unclear about my what should I do to solve the issue.
I am just a beginner and lack in-dept knowledge about ES.
Can you please give me exact steps to avoid the problem that I am facing.

8

Based on the conversation above..

  • Shutdown your elastic search server right away- you maybe compromised.
  • Figure out how to remove it's exposure to the internet.
  • Iptables or any kind of firewall will do
  • Read the blog above
  • you may have to check your system for "suspicious files" as perl is not from ES and if you are not running it. (SOMEONE IS and they might not be good people)
9

Still I am unclear about my what should I do to solve the issue.
I am just a beginner and lack in-dept knowledge about ES.
Can you please give me exact steps to avoid the problem that I am facing.

Also in order to run ctx._source.Added+=1 I need to enable dynamic scripting is there any alternative to it?