# High cpu load elasticsearch on logstash output **URL:** \ **Category:** Elasticsearch\ **Created:** [December 15, 2018, 8:23pm UTC](https://discuss.elastic.co/t/high-cpu-load-elasticsearch-on-logstash-output/161009 "2018-12-15T20:23:51Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![tim.van.rooijen](https://avatars.discourse-cdn.com/v4/letter/t/a4c791/32.png) [@tim.van.rooijen](https://discuss.elastic.co/u/tim.van.rooijen)\ **Post date:** [December 15, 2018, 8:23pm UTC](https://discuss.elastic.co/t/high-cpu-load-elasticsearch-on-logstash-output/161009/1 "2018-12-15T20:23:51Z")
Hi, Beginner with the elk stack. Have an api thats sends messages to rabbit mq. Logstash reads the messages and adds them to elasticsearch. When the api is under peak load the cpu load on elasticsearch is really high. Its a single node cluster but it has some good hardware. My logstash output looks like this: For me it's not completely clear if logstash is sending these messages in bulk to elasticsearch or not. Anybody could give me some tips on how to improve performance? elasticsearch { index =\> "api\_requests-%{+YYYY.MM.dd}" document\_id =\> "%{id}" hosts =\> "elastic-logging-01:9200" document\_type =\> "doc" template =\> "/opt/plugin/MappingTemplates/api\_requests.json" template\_name =\> "api\_requests-template" manage\_template =\> true template\_overwrite =\> true } ---
**Author:** ![rugenl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rugenl/32/12887_2.png) [@rugenl](https://discuss.elastic.co/u/rugenl)\ **Post date:** [December 15, 2018, 11:47pm UTC](https://discuss.elastic.co/t/high-cpu-load-elasticsearch-on-logstash-output/161009/2 "2018-12-15T23:47:30Z")
I see you are sending an ID, that could be a cause, see [Bad bulk performance with self-generated id](https://discuss.elastic.co/t/bad-bulk-performance-with-self-generated-id/103344) ---
**Author:** ![tim.van.rooijen](https://avatars.discourse-cdn.com/v4/letter/t/a4c791/32.png) [@tim.van.rooijen](https://discuss.elastic.co/u/tim.van.rooijen)\ **Post date:** [December 17, 2018, 7:53am UTC](https://discuss.elastic.co/t/high-cpu-load-elasticsearch-on-logstash-output/161009/3 "2018-12-17T07:53:44Z")
Thanks for your reply. Sounds like a good point. Moving this into production today. Hopefully this will solve our problem ---
**Author:** ![tim.van.rooijen](https://avatars.discourse-cdn.com/v4/letter/t/a4c791/32.png) [@tim.van.rooijen](https://discuss.elastic.co/u/tim.van.rooijen)\ **Post date:** [December 17, 2018, 9:01am UTC](https://discuss.elastic.co/t/high-cpu-load-elasticsearch-on-logstash-output/161009/4 "2018-12-17T09:01:58Z")
Moved it into production. But sadly not improvements in cpu usage elasticsearch. Any other suggestions? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 17, 2018, 9:05am UTC](https://discuss.elastic.co/t/high-cpu-load-elasticsearch-on-logstash-output/161009/5 "2018-12-17T09:05:56Z")
Indexing can be quite CPU intensive. What is the average size of your documents? What indexing throughput are you seeing? What is the specification of the hardware your cluster is running on? Is there anything in the logs around long or frequent GC? ---
**Author:** ![tim.van.rooijen](https://avatars.discourse-cdn.com/v4/letter/t/a4c791/32.png) [@tim.van.rooijen](https://discuss.elastic.co/u/tim.van.rooijen)\ **Post date:** [December 17, 2018, 9:36am UTC](https://discuss.elastic.co/t/high-cpu-load-elasticsearch-on-logstash-output/161009/6 "2018-12-17T09:36:43Z")
Arround 50 messages a second. Doc size is arround 1000 bytes Elasticsearch has 4 cpu cores 3.2 g and 7 gigs of memory Find nothing strange in logs elasticsearch or logstash ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 17, 2018, 9:39am UTC](https://discuss.elastic.co/t/high-cpu-load-elasticsearch-on-logstash-output/161009/7 "2018-12-17T09:39:13Z")
You mention that the CPU load is really high. How high is that? Do you have monitoring installed so you can see what is going on? ---
**Author:** ![tim.van.rooijen](https://avatars.discourse-cdn.com/v4/letter/t/a4c791/32.png) [@tim.van.rooijen](https://discuss.elastic.co/u/tim.van.rooijen)\ **Post date:** [December 17, 2018, 9:51am UTC](https://discuss.elastic.co/t/high-cpu-load-elasticsearch-on-logstash-output/161009/8 "2018-12-17T09:51:32Z")
Under peak load cpu goes to 100%. It still cannot keep up with rabbit and logstash starts throwing exceptions. Will install monitoring later today ---
**Author:** ![rugenl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rugenl/32/12887_2.png) [@rugenl](https://discuss.elastic.co/u/rugenl)\ **Post date:** [December 18, 2018, 1:58am UTC](https://discuss.elastic.co/t/high-cpu-load-elasticsearch-on-logstash-output/161009/9 "2018-12-18T01:58:27Z")
Does your template define all fields or are you doing dynamic mapping? The mapping that analyzes text fields and maps them as keyword can do more than you sometimes need. ---
**Author:** ![tim.van.rooijen](https://avatars.discourse-cdn.com/v4/letter/t/a4c791/32.png) [@tim.van.rooijen](https://discuss.elastic.co/u/tim.van.rooijen)\ **Post date:** [December 18, 2018, 9:33am UTC](https://discuss.elastic.co/t/high-cpu-load-elasticsearch-on-logstash-output/161009/10 "2018-12-18T09:33:36Z")
The mapping template contains: "dynamic": "false" So this should be oke. @rugenl Analysed the logs from yesterday. And removing the id did give us some improvements. Cpu load is still high but its better than before, so thanks for that one!! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 15, 2019, 9:36am UTC](https://discuss.elastic.co/t/high-cpu-load-elasticsearch-on-logstash-output/161009/11 "2019-01-15T09:36:02Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.