# How to gain ingestion rate **URL:** \ **Category:** Elasticsearch\ **Created:** [May 8, 2016, 7:22am UTC](https://discuss.elastic.co/t/how-to-gain-ingestion-rate/49476 "2016-05-08T07:22:47Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![Marcin\_Kubica](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_kubica/32/5614_2.png) [@Marcin\_Kubica](https://discuss.elastic.co/u/Marcin_Kubica)\ **Post date:** [May 8, 2016, 7:22am UTC](https://discuss.elastic.co/t/how-to-gain-ingestion-rate/49476/1 "2016-05-08T07:22:47Z")
Hi, trying to pass 10k doc/s on a 9 node 8c/32g ram cluster Dataset in totall can reach 400GB Started small with 6 shards no replicas and can't reach above 10k docs/s What am I missing? Should I allocate a shard per core? Long index refresh aslo doesnt seem to change anything. Cheers Marcin ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 8, 2016, 7:36am UTC](https://discuss.elastic.co/t/how-to-gain-ingestion-rate/49476/2 "2016-05-08T07:36:52Z")
Are you using bulk? What client are you using for ingestion? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 8, 2016, 8:32am UTC](https://discuss.elastic.co/t/how-to-gain-ingestion-rate/49476/3 "2016-05-08T08:32:07Z")
What is the average size of your documents? Which version of Elasticsearch are you using? Are you using time-based indices? ---
**Author:** ![Marcin\_Kubica](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_kubica/32/5614_2.png) [@Marcin\_Kubica](https://discuss.elastic.co/u/Marcin_Kubica)\ **Post date:** [May 8, 2016, 8:41am UTC](https://discuss.elastic.co/t/how-to-gain-ingestion-rate/49476/4 "2016-05-08T08:41:00Z")
Hi Mark Bulk and with logstash. Cheers Marcin ---
**Author:** ![Marcin\_Kubica](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_kubica/32/5614_2.png) [@Marcin\_Kubica](https://discuss.elastic.co/u/Marcin_Kubica)\ **Post date:** [May 8, 2016, 8:59am UTC](https://discuss.elastic.co/t/how-to-gain-ingestion-rate/49476/6 "2016-05-08T08:59:52Z")
Hi Chris It's an Apache style log, like your usual http server log with URL, response code and few tags. We use @timestamp as usual. Marcin ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 8, 2016, 9:33am UTC](https://discuss.elastic.co/t/how-to-gain-ingestion-rate/49476/7 "2016-05-08T09:33:40Z")
For such small documents that sounds very low given the size of the cluster. How many Logstash instances do have indexing into the cluster? How are these configured? ---
**Author:** ![Marcin\_Kubica](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_kubica/32/5614_2.png) [@Marcin\_Kubica](https://discuss.elastic.co/u/Marcin_Kubica)\ **Post date:** [May 9, 2016, 1:50am UTC](https://discuss.elastic.co/t/how-to-gain-ingestion-rate/49476/8 "2016-05-09T01:50:53Z")
Hi Christian I've taken a step back and graphed logstash instances and it seems it's simply Redis which is limiting currently. We seem to be getting say around 1.5k average per logstash instance. However seems just reached Redis limit. Looking into that now. ![](https://us1.discourse-cdn.com/elastic/original/2X/9/98ac7295087033e6c7dbc4845dcc36316ffad001.png) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 9, 2016, 5:06am UTC](https://discuss.elastic.co/t/how-to-gain-ingestion-rate/49476/9 "2016-05-09T05:06:44Z")
What does your logstash configuration look like? ---
**Author:** ![Marcin\_Kubica](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_kubica/32/5614_2.png) [@Marcin\_Kubica](https://discuss.elastic.co/u/Marcin_Kubica)\ **Post date:** [May 9, 2016, 5:41am UTC](https://discuss.elastic.co/t/how-to-gain-ingestion-rate/49476/10 "2016-05-09T05:41:37Z")
Hi Christian Usually like this [https://github.com/mutl3y/logstash/blob/master/roles/logstash/templates/lds\_logs.conf.j2](https://github.com/mutl3y/logstash/blob/master/roles/logstash/templates/lds_logs.conf.j2) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 9, 2016, 8:22am UTC](https://discuss.elastic.co/t/how-to-gain-ingestion-rate/49476/11 "2016-05-09T08:22:57Z")
Why are you using a ruby filter to calculate MD5 hash of the message field when the [fingerprint filter plugin](https://www.elastic.co/guide/en/logstash/current/plugins-filters-fingerprint.html) exists for this purpose? I would expect this to be more efficient than to invoke the ruby script for each event, but have not benchmarked it. ---
**Author:** ![Marcin\_Kubica](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_kubica/32/5614_2.png) [@Marcin\_Kubica](https://discuss.elastic.co/u/Marcin_Kubica)\ **Post date:** [May 9, 2016, 9:56am UTC](https://discuss.elastic.co/t/how-to-gain-ingestion-rate/49476/12 "2016-05-09T09:56:00Z")
Will try that fingerprint but found other clue. If I play 4 processors and reach about 10k ingest, if I'm adding more it has no effect. Also, when the above four are in play and I spawn 2 totally different ingesters with stdin input again all of them seem to kneel down at 10k. The different two I mention each can play at 2k. So clearly not the each single logstash node problem. Currently using 16 elastic nodes ( 9 spec as mentioned ) and 7 other a little less spec. Any idea what's the cause? Almost like elastic would have ingest rate hard limit to 10k on cluster. Is that the case? Many thanks Marcin ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 9, 2016, 10:16am UTC](https://discuss.elastic.co/t/how-to-gain-ingestion-rate/49476/13 "2016-05-09T10:16:04Z")
Which version of Logstash are you using? How many workers do you have configured? In order to find out where the bottleneck is I would recommend starting from Elasticsearch and work up the pipeline. Set up a simple Logstash config that reads from a file and index into a test index. See if you are able to push Elasticsearch harder and achieve a better indexing rate this way. If this is the case, feed Logstash from a file (in order to rule out Redis dequeueing) and see what it is able to do. Continue up the pipeline while optimising settings until you find the bottleneck. ---
**Author:** ![Marcin\_Kubica](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_kubica/32/5614_2.png) [@Marcin\_Kubica](https://discuss.elastic.co/u/Marcin_Kubica)\ **Post date:** [May 9, 2016, 10:23am UTC](https://discuss.elastic.co/t/how-to-gain-ingestion-rate/49476/14 "2016-05-09T10:23:10Z")
Thanks, would you advise single logstash instance per file to ingest and with single shard? And maybe route each index per node? And then just multiplying that? Or maybe other way? Many thanks again Marcin ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 9, 2016, 10:27am UTC](https://discuss.elastic.co/t/how-to-gain-ingestion-rate/49476/15 "2016-05-09T10:27:15Z")
I would set up a test index with as many shards as there are data nodes and add as many Logstash nodes as needed until you have been able to verify whether Elasticsearch is the bottleneck or not. ---
**Author:** ![Marcin\_Kubica](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_kubica/32/5614_2.png) [@Marcin\_Kubica](https://discuss.elastic.co/u/Marcin_Kubica)\ **Post date:** [May 9, 2016, 10:28am UTC](https://discuss.elastic.co/t/how-to-gain-ingestion-rate/49476/16 "2016-05-09T10:28:47Z")
Thanks! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:53pm UTC](https://discuss.elastic.co/t/how-to-gain-ingestion-rate/49476/17 "2017-07-05T22:53:02Z")