# Bottleneck while inputting data into the elasticsearch **URL:** \ **Category:** Logstash\ **Created:** [December 1, 2016, 8:58am UTC](https://discuss.elastic.co/t/bottleneck-while-inputting-data-into-the-elasticsearch/67735 "2016-12-01T08:58:34Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Mayank\_Agrawal](https://avatars.discourse-cdn.com/v4/letter/m/3ab097/32.png) [@Mayank\_Agrawal](https://discuss.elastic.co/u/Mayank_Agrawal)\ **Post date:** [December 1, 2016, 8:58am UTC](https://discuss.elastic.co/t/bottleneck-while-inputting-data-into-the-elasticsearch/67735/1 "2016-12-01T08:58:34Z")
Hi, I am using elasticsearch 5.0.1 and logstash 5.0.0. For 1GB data, logstash is easily parsing the logs in half an hr (approx) if I am not redirecting the output to ES. With ES in output, the time taken to parse the same data is around 3.5-4 hours. How do I reduce the bottleneck during the insertion of data in elasticsearch ??? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 1, 2016, 9:32am UTC](https://discuss.elastic.co/t/bottleneck-while-inputting-data-into-the-elasticsearch/67735/2 "2016-12-01T09:32:39Z")
What is the specification of your Elasticsearch cluster? Have you looked to identify what is limiting Elasticsearch performance? Is CPU saturated? Do you see a lot of IO wait? Is there evidence of a lot of GC in the logs? ---
**Author:** ![Mayank\_Agrawal](https://avatars.discourse-cdn.com/v4/letter/m/3ab097/32.png) [@Mayank\_Agrawal](https://discuss.elastic.co/u/Mayank_Agrawal)\ **Post date:** [December 1, 2016, 9:38am UTC](https://discuss.elastic.co/t/bottleneck-while-inputting-data-into-the-elasticsearch/67735/3 "2016-12-01T09:38:53Z")
Hi Christian, I am using the default settings of elasticsearch 5.0.1. In the logstash config file, I have the following setting in output plugin: ``` elasticsearch { hosts => ["localhost:9200"] index => "eaxmple-test1" } ``` How do I check its performance (elasticsearch) ? Sorry, I am new to logstash and elasticsearch.. Logstash shows following config (default I guess): {"id"=\>"main", "pipeline.workers"=\>4, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>5, "pipeline.max\_inflight"=\>500} The machine on which I tested it has the following config: Processor: Intel Core i3-4130 CPU @ 3.40GHz RAM: 4 GB System type: 64bit ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 1, 2016, 9:54am UTC](https://discuss.elastic.co/t/bottleneck-while-inputting-data-into-the-elasticsearch/67735/4 "2016-12-01T09:54:12Z")
Look at the operating system level to see if CPU is fully utilised. You only have 2 physical cores, which makes it likely that is the bottleneck. What is the size of your events? Is Logstash also running on the same host? ---
**Author:** ![Mayank\_Agrawal](https://avatars.discourse-cdn.com/v4/letter/m/3ab097/32.png) [@Mayank\_Agrawal](https://discuss.elastic.co/u/Mayank_Agrawal)\ **Post date:** [December 1, 2016, 10:19am UTC](https://discuss.elastic.co/t/bottleneck-while-inputting-data-into-the-elasticsearch/67735/5 "2016-12-01T10:19:35Z")
OS level performance: 1. CPU : varying between 11%-35% 2. Memory: stable at 69% 3. Disk: Stable at 99% 4. Network: 0% Yes, logstash is also running on the same host. FYI, 1 GB data that I mentioned in the post was divided into ~100 MB files. > [@Christian\_Dahlqvist](#): > > What is the size of your events? What do you mean by 'size of the events' ?? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 1, 2016, 10:35am UTC](https://discuss.elastic.co/t/bottleneck-while-inputting-data-into-the-elasticsearch/67735/6 "2016-12-01T10:35:02Z")
Unless you have exceptionally slow disk, I would expect CPU to be the limit if running both Logstash and Elasticsearch on the same host. Have you set the number of workers in the Logstash elasticsearch output as outlined in the [performance troubleshooting guide](https://www.elastic.co/guide/en/logstash/current/performance-troubleshooting.html)? What type of data are you indexing? How large are the records? ---
**Author:** ![Mayank\_Agrawal](https://avatars.discourse-cdn.com/v4/letter/m/3ab097/32.png) [@Mayank\_Agrawal](https://discuss.elastic.co/u/Mayank_Agrawal)\ **Post date:** [December 1, 2016, 10:56am UTC](https://discuss.elastic.co/t/bottleneck-while-inputting-data-into-the-elasticsearch/67735/7 "2016-12-01T10:56:55Z")
> [@Christian\_Dahlqvist](#): > > Have you set the number of workers in the Logstash elasticsearch output as outlined in the performance troubleshooting guide? As per the [documentation](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-workers) of elasticsearch plugin, output workers are no longer supported. > [@Christian\_Dahlqvist](#): > > What type of data are you indexing? How large are the records? Data contains logs in plain text. From each log, few key-value pairs are extracted using logstash filters. Each log message can be between 100-1500 characters. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 29, 2016, 10:57am UTC](https://discuss.elastic.co/t/bottleneck-while-inputting-data-into-the-elasticsearch/67735/8 "2016-12-29T10:57:12Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.