High CPU Utilized by JAVA in single node ELK #31089

Elastic Stack Elasticsearch
1

We are experiencing high CPU utilization when starting the elasticsearch & logstash services simultaneously.
The JAVA process is occupying more than 300% CPU.

The elasticsearch holds some multiple JAVA threads and makes the CPU reaches to max utilization.

Elasticsearch version (bin/elasticsearch --version): 2.3.4

JVM version (java -version): 1.8.0

OS version (uname -a if on a Unix-like system): RHEL 7.2

image
image.png1476×145 55.6 KB

2

Team,

Can you help on this?

We are at critical state due to this CPU issue.

3

Do you have monitoring installed? How much indexing and querying is the node handling when the CPU usage goes up?

4

Hi,

We have nagios installed for monitoring the load . the load keeps on fluctuating and its has lot of threads running at the backend.

When I query the indices , I am getting the below o/p with lot of indices in open state.

image
image.png1165×976 177 KB

Please assist, since I am very new to ELK

5

You seem to have a lot of small shards, so please read this blog post for some guidance on shards and sharding practices. When you look at monitoring and amount of data indexed, do you see any correlation between indexed volume and CPU usage?

6

Yes Christian. Initial days the CPU usage was very well under control.
Now, as the data indices are growing it starts affecting my CPU usage.

Also, I could suspect that my indices are creating duplicates as the older date indices are also growing on day to day basis as well.

Please advise on how to avoid duplicate indices from getting created .

7

Have a look at your ingest pipeline and make sure it is configured correctly so it does not resend old data.

8

Could you please be more specific on which configuration file to be checked?

9

I do not know anything about how you are ingesting data, so it is hard to be more specific. How are you ingesting data into Elasticsearch?

10

  • Using RabbitMQ as buffer

  • Configured Logstash to push logs to RabbitMQ

  • Pulling logs from RabbitMQ and indexing

Is any other information required?

11

@Christian_Dahlqvist Please find below the logstash.conf

input 
{ rabbitmq { 
   host => "10.x.x.x" 
   port => 5670 
   ssl_certificate_password => 'xxxxx' 
   ssl_certificate_path => Vetc/logstash/ssl_cert/client/keycert.p12' 
   ssl_version => "TLSv1.2" 
   ssl => true 
   durable => false 
   exchange => "elasticsearch-exchange" 
   queue => "indexer-queue" 
   vhost => "es_vhost" 
   key => "logstash-routing_key" 
   user => "xxxx" 
   password => "xxxx" 
   heartbeat => "15" 
   automatic_recovery => false 
   connect_retry_interval => 1 
   exchange type => "direct" 
   }
} 

output { 

  elasticsearch { 
    hosts => ["localhostm] 
	index => "immediate-%(+YYYY.MM.dd}"
	} 
}

input { 

  file { 
    path => "/var/log/messages" 
    start position => "end" 
	}
 
}
filter { 
  grok { 
        match => "message" => "%{SYSLOGTIMESTAMP:system_auth_timestamp} %{SYSLOGHOST:system_auth_hostname} %{GREEDYDATA:command_issued}: %{GREEDYDATA:message}" } 
		add tag => "syslog"
		} 
}

output {
	rabbitmq { 
		exchange => "elasticsearch-exchange" 
		exchange_type => "direct" 
		key => "logstash-routing_key" 
		ssl => true 
		durable => false 
		ssl_certificate_password => 'xxxxx' 
		ssl_certificate_path => 'etc/logstash/ssl_cert/client/keycert.p12' 
		ssl_version => "TLSv1.2" 
		host => "10.x.x.x" 
		vhost => "es_vhost" 
		persistent => true 
		passive => true 
		port => "5671" 
		user => "xxxx" 
		password => "xxxx"
		}
}
12

I do unfortunately not have much experience with RabbitMQ, so may not be able to help much.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.