# Logstash performance for indexing to ES
**URL:** https://discuss.elastic.co/t/logstash-performance-for-indexing-to-es/79155
**Category:** Logstash
**Created:** [March 19, 2017, 1:08am UTC](https://discuss.elastic.co/t/logstash-performance-for-indexing-to-es/79155 "2017-03-19T01:08:53Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@Sharat\_Jagannath](https://discuss.elastic.co/u/Sharat_Jagannath)
#### Post date: [March 19, 2017, 1:08am UTC](https://discuss.elastic.co/t/logstash-performance-for-indexing-to-es/79155/1 "2017-03-19T01:08:53Z")
I'm trying to index docs to elasticsearch from logstash with kafka input. I see the indexing takes a long time and i am looking to index around 3 million documents as fast as possible. My logstash conf is -
```
input {
kafka {
bootstrap_servers => "kafka2.dev:9092"
topics => ["READY_FOR_INDEX","INDEX_CSV"]
codec => json
consumer_threads => 8
}
}
output {
stdout { codec => rubydebug }
if [type] == "fact" or [type] == "dimension" {
elasticsearch {
index => "%{index}"
document_id => "%{id}"
hosts => "xyz.amazonaws.com:9200"
flush_size=>100000
}
}else {
elasticsearch {
index => "%{index}"
document_id => "%{id}"
hosts => "xyz.amazonaws.com:9200"
flush_size=>100000
}
}
}
```
My logstash is on aws machine with 32 gb RAM and 8 cores.
Right now just for 1000 documents it takes around 4 minutes. By that calculation it would take really long to index 3 million.
I was under the assumption that logstash did ES bulk indexing.
Please help.
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [March 20, 2017, 11:18am UTC](https://discuss.elastic.co/t/logstash-performance-for-indexing-to-es/79155/2 "2017-03-20T11:18:21Z")
> Right now just for 1000 documents it takes around 4 minutes.
Four minutes for 1000 docs? That's ridiculously slow. Logstash itself shouldn't have any problems processing hundreds of events per second.
> I was under the assumption that logstash did ES bulk indexing.
It does.
Have you measured what the bottleneck is? What event rate do you get with a kafka input and e.g. a file output? What event rate do you get against ES without using Logstash?
---
### Author:  [@Sharat\_Jagannath](https://discuss.elastic.co/u/Sharat_Jagannath)
#### Post date: [March 20, 2017, 6:51pm UTC](https://discuss.elastic.co/t/logstash-performance-for-indexing-to-es/79155/3 "2017-03-20T18:51:00Z")
I think kafka has the enable\_metrics field set to true in logstash by default. But how do i see where the metrics are stored? I guess thats how i get how fast the events are processed.
Thanks
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [March 20, 2017, 7:14pm UTC](https://discuss.elastic.co/t/logstash-performance-for-indexing-to-es/79155/4 "2017-03-20T19:14:30Z")
You should be able to use Logstash's HTTP API for that.
---
### Author:  [@Sharat\_Jagannath](https://discuss.elastic.co/u/Sharat_Jagannath)
#### Post date: [March 21, 2017, 12:30am UTC](https://discuss.elastic.co/t/logstash-performance-for-indexing-to-es/79155/5 "2017-03-21T00:30:36Z")
Does the thruput also depend on how fast we write to kafka?
Example if i write to kafka serially or parallely?
Thanks
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [March 21, 2017, 6:09am UTC](https://discuss.elastic.co/t/logstash-performance-for-indexing-to-es/79155/6 "2017-03-21T06:09:17Z")
Unless you have a backlog in Kafka the read rate can obviously not be greater than the write rate. A sufficiently high inflow of new events could also affect the read rate if the server is heavily loaded, but I don't really know enough about Kafka to say if it employs any countermeasures against that (like RabbitMQ's throttling of publishers when the consumers can't keep up).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 18, 2017, 6:09am UTC](https://discuss.elastic.co/t/logstash-performance-for-indexing-to-es/79155/7 "2017-04-18T06:09:32Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.