# Logstash is very slow in sending the data to elasticsearch
**URL:** https://discuss.elastic.co/t/logstash-is-very-slow-in-sending-the-data-to-elasticsearch/72130
**Category:** Logstash
**Created:** [January 19, 2017, 8:15am UTC](https://discuss.elastic.co/t/logstash-is-very-slow-in-sending-the-data-to-elasticsearch/72130 "2017-01-19T08:15:41Z")
**Posts on this page:** 19
**Page:** 1
### Author:  [@anisen](https://discuss.elastic.co/u/anisen)
#### Post date: [January 19, 2017, 8:15am UTC](https://discuss.elastic.co/t/logstash-is-very-slow-in-sending-the-data-to-elasticsearch/72130/1 "2017-01-19T08:15:41Z")
Hi,
I am parsing an IIS log-file which is in below format:
2014-07-12:00:08:54 100.200.50.0 GET /mypath/mypage.asmx - 80 - 100.100.50.0 Mozilla/4.1+(compatible;+MSIE+4.01;+Windows+NT;+MS+Search+8.0+Robot) - 403 0 0 39
And, my logstash config file looks like this:
input {
file {
path =\> "D:/myLogFile.log"
type =\> "iis-log"
start\_position =\> "beginning"
}
}
filter {
grok {
match =\> {
"message" =\> '%{DATA:timestamp} %{IPORHOST:clientip} %{NOTSPACE:method} %{NOTSPACE:uri} %{NOTSPACE:csuriquery} %{NOTSPACE:port} %{NOTSPACE:username} %{NOTSPACE:serverip} %{NOTSPACE:agent} %{NOTSPACE:referrer} %{NOTSPACE:status} %{NOTSPACE:sub\_status} %{NOTSPACE:win\_status} %{NOTSPACE:responsetime}'
}
}
date {
match =\> ["timestamp", "YYYY-mm-dd:HH:mm:ss"]
locale =\> en
}
}
output {
elasticsearch {
action =\> "index"
hosts =\> "localhost"
index =\> "myindex"
}
}
I am processing around 10 lakh data, for which it is taking around 2 hours, which is much much slower than what I expected.
Kindly suggest how can I increase the performance/ tune the above config file/filter.
Thanks in advance.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [January 19, 2017, 8:27am UTC](https://discuss.elastic.co/t/logstash-is-very-slow-in-sending-the-data-to-elasticsearch/72130/2 "2017-01-19T08:27:44Z")
Depending on which version of Logstash you are using, recommended tuning will differ as the pipeline has gone through changes. I would however recommend increasing the number of [workers](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-workers) for the elastic search output a bit as a first step. Start setting this to a reasonably low number and increase this slowly until it no longer improves throughput. A good starting point may be the number of worker threads you have or the number of cores available one the Logstash host.
---
### Author:  [@Nitz](https://discuss.elastic.co/u/Nitz)
#### Post date: [January 19, 2017, 8:38am UTC](https://discuss.elastic.co/t/logstash-is-very-slow-in-sending-the-data-to-elasticsearch/72130/3 "2017-01-19T08:38:15Z")
+1
We experienced it too. Increasing number of workers didn't helped much.
Would like to get further assistance on this one.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [January 19, 2017, 8:43am UTC](https://discuss.elastic.co/t/logstash-is-very-slow-in-sending-the-data-to-elasticsearch/72130/4 "2017-01-19T08:43:14Z")
Which version of Logstash are you using?
---
### Author:  [@anisen](https://discuss.elastic.co/u/anisen)
#### Post date: [January 19, 2017, 8:55am UTC](https://discuss.elastic.co/t/logstash-is-very-slow-in-sending-the-data-to-elasticsearch/72130/5 "2017-01-19T08:55:43Z")
Logstash and Elasticsearch both are of 5.1.1 version.
Moreover, it seems workers are depricated in logstash 5.1.1.
Any alternate to this?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [January 19, 2017, 9:06am UTC](https://discuss.elastic.co/t/logstash-is-very-slow-in-sending-the-data-to-elasticsearch/72130/6 "2017-01-19T09:06:37Z")
The [documentation](https://www.elastic.co/guide/en/logstash/current/performance-troubleshooting.html) provides some good guidance on troubleshooting performance. Increasing the number of workers in the Elasticsearch output is one of the things discussed there.
What is the hardware specification of you Logstash host and Elasticsearch cluster?
---
### Author:  [@anisen](https://discuss.elastic.co/u/anisen)
#### Post date: [January 19, 2017, 9:13am UTC](https://discuss.elastic.co/t/logstash-is-very-slow-in-sending-the-data-to-elasticsearch/72130/7 "2017-01-19T09:13:58Z")
Could you please let me know how I can increase the number of workers?
I had this config:
output {
elasticsearch {
action =\> "index"
hosts =\> "localhost"
index =\> "myindex"
workers =\> 1
}
}
But, increasing the number of workers to 2 or more, giving the below error message:
"You are using a plugin that doesn't support workers but have set the workers value explicitly! This plugin uses the shared and doesn't need this option"
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [January 19, 2017, 9:50am UTC](https://discuss.elastic.co/t/logstash-is-very-slow-in-sending-the-data-to-elasticsearch/72130/8 "2017-01-19T09:50:02Z")
The Elasticsearch output in 5.1.1 apparently no longer need the workers parameter, as this is handled automatically as it is thread-safe. This could however be better documented in my opinion, so I will open an issue.
---
### Author:  [@guyboertje](https://discuss.elastic.co/u/guyboertje)
#### Post date: [January 19, 2017, 9:58am UTC](https://discuss.elastic.co/t/logstash-is-very-slow-in-sending-the-data-to-elasticsearch/72130/9 "2017-01-19T09:58:56Z")
What is 10 lakh data?
---
### Author:  [@guyboertje](https://discuss.elastic.co/u/guyboertje)
#### Post date: [January 19, 2017, 9:59am UTC](https://discuss.elastic.co/t/logstash-is-very-slow-in-sending-the-data-to-elasticsearch/72130/10 "2017-01-19T09:59:52Z")
What is the spec of the machine you are running Logstash on?
---
### Author:  [@guyboertje](https://discuss.elastic.co/u/guyboertje)
#### Post date: [January 19, 2017, 10:00am UTC](https://discuss.elastic.co/t/logstash-is-very-slow-in-sending-the-data-to-elasticsearch/72130/11 "2017-01-19T10:00:31Z")
Are you running Elasticsearch and Logstash on the same machine?
---
### Author:  [@guyboertje](https://discuss.elastic.co/u/guyboertje)
#### Post date: [January 19, 2017, 10:03am UTC](https://discuss.elastic.co/t/logstash-is-very-slow-in-sending-the-data-to-elasticsearch/72130/12 "2017-01-19T10:03:38Z")
Try anchoring your grok pattern to the beginning of the string e.g.
```ruby
grok {
match => { "message" => '^%{DATA:timestamp} %{IPORHOST:clientip} %{NOTSPACE:method} %{NOTSPACE:uri} %{NOTSPACE:csuriquery} %{NOTSPACE:port} %{NOTSPACE:username} %{NOTSPACE:serverip} %{NOTSPACE:agent} %{NOTSPACE:referrer} %{NOTSPACE:status} %{NOTSPACE:sub_status} %{NOTSPACE:win_status} %{NOTSPACE:responsetime}' }
}
```
add a `^` to the beginning of the pattern.
See [https://www.elastic.co/blog/do-you-grok-grok](https://www.elastic.co/blog/do-you-grok-grok)
---
### Author:  [@anisen](https://discuss.elastic.co/u/anisen)
#### Post date: [January 19, 2017, 10:06am UTC](https://discuss.elastic.co/t/logstash-is-very-slow-in-sending-the-data-to-elasticsearch/72130/13 "2017-01-19T10:06:56Z")
Yes. I'm running Elasticsearch and Logstash on the same machine, which has the below configuration:
Processor: AMD Athlon(tm) || X2 245 Processor @2.90 GHz
RAM : 8 GB
Tried with anchoring as well, but no appreciable difference in performance.
---
### Author:  [@anisen](https://discuss.elastic.co/u/anisen)
#### Post date: [January 19, 2017, 11:52am UTC](https://discuss.elastic.co/t/logstash-is-very-slow-in-sending-the-data-to-elasticsearch/72130/14 "2017-01-19T11:52:20Z")
1 million lines of logs.
---
### Author:  [@guyboertje](https://discuss.elastic.co/u/guyboertje)
#### Post date: [January 19, 2017, 12:22pm UTC](https://discuss.elastic.co/t/logstash-is-very-slow-in-sending-the-data-to-elasticsearch/72130/15 "2017-01-19T12:22:01Z")
Are you getting events tagged with `_grokparsefailure`?
---
### Author:  [@anisen](https://discuss.elastic.co/u/anisen)
#### Post date: [January 19, 2017, 12:25pm UTC](https://discuss.elastic.co/t/logstash-is-very-slow-in-sending-the-data-to-elasticsearch/72130/16 "2017-01-19T12:25:17Z")
I [Checked.No](http://Checked.No) failures.
---
### Author:  [@guyboertje](https://discuss.elastic.co/u/guyboertje)
#### Post date: [January 19, 2017, 12:33pm UTC](https://discuss.elastic.co/t/logstash-is-very-slow-in-sending-the-data-to-elasticsearch/72130/17 "2017-01-19T12:33:27Z")
1000000 / 2 / 60 / 60 is 138 events per second. You should be seeing ~2000 to 4000 events per second.
Try excluding Elasticsearch by using the `stdout` output with the `dots` codec.
You should see the dots stop after about 4 minutes - if so then ES is the bottleneck.
---
### Author:  [@anisen](https://discuss.elastic.co/u/anisen)
#### Post date: [January 24, 2017, 11:14am UTC](https://discuss.elastic.co/t/logstash-is-very-slow-in-sending-the-data-to-elasticsearch/72130/18 "2017-01-24T11:14:56Z")
> [@guyboertje](#):
>
> 2000 to 4000
Tried removing elasticsearch and using dots codec, it is taking 7 mins.
But, I need the data in Elasticsearch, so sending data to ES in different machine instead of localhost:
output {
elasticsearch {
action =\> "index"
hosts =\> "100.100.0.10:9200"
index =\> "myindex"
}
}
Still taking a lot of time (around 90 mins). Any suggestion how to improve this?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 21, 2017, 11:14am UTC](https://discuss.elastic.co/t/logstash-is-very-slow-in-sending-the-data-to-elasticsearch/72130/19 "2017-02-21T11:14:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.