# Increase insert and search rate in elastic search

**URL:** <https://discuss.elastic.co/t/increase-insert-and-search-rate-in-elastic-search/254963>\
**Category:** Elasticsearch\
**Created:** [November 10, 2020, 6:29pm UTC](https://discuss.elastic.co/t/increase-insert-and-search-rate-in-elastic-search/254963 "2020-11-10T18:29:16Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sherrie](https://avatars.discourse-cdn.com/v4/letter/s/a3d4f5/32.png) [@Sherrie](https://discuss.elastic.co/u/Sherrie)\
**Post date:** [November 10, 2020, 6:29pm UTC](https://discuss.elastic.co/t/increase-insert-and-search-rate-in-elastic-search/254963/1 "2020-11-10T18:29:16Z")

</div>

Hi,

We are using a JAVA API to insert 100k records from a kafka topic to elastic search endpoints to a particular index. The insert rate for 100k records takes upto 30 minutes. Whereas the same 100k records can be inserted in DSE (Cassandra+Solr) in 5-7minutes. Is there a way to optimize the insert/search rate using any index optimization ?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 10, 2020, 6:51pm UTC](https://discuss.elastic.co/t/increase-insert-and-search-rate-in-elastic-search/254963/2 "2020-11-10T18:51:19Z")

</div>

Have you gone through [these guidelines](https://www.elastic.co/guide/en/elasticsearch/reference/7.9/tune-for-indexing-speed.html)? What is the size of your documents? How are you indexing the data? What bulk size are you using? How many concurrent clients are indexing? What is the hardware specification of the cluster?

---

<div class="post-metadata">

**Author:** ![Sherrie](https://avatars.discourse-cdn.com/v4/letter/s/a3d4f5/32.png) [@Sherrie](https://discuss.elastic.co/u/Sherrie)\
**Post date:** [November 11, 2020, 3:15pm UTC](https://discuss.elastic.co/t/increase-insert-and-search-rate-in-elastic-search/254963/3 "2020-11-11T15:15:10Z")

</div>

> [@Sherrie](#):
>
> Whereas the same 100k records can be inserted in DSE (Cassandra+Solr) in 5-7minutes. Is there a way to optimize the insert/search rate using any index optimization ?

Hi Christian,  
These are 1kb documents. They are ingested into kafka topic via JMeter. We are trying with the increased refresh\_interval also. We have tried with 3 concurrent clients (3 instances)  
The hardware specification is 4 core CPU - 10 node clusters.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 11, 2020, 3:29pm UTC](https://discuss.elastic.co/t/increase-insert-and-search-rate-in-elastic-search/254963/4 "2020-11-11T15:29:10Z")

</div>

How are you getting the data from Kafka into Elasticsearch? Is the hardware used in the two environments the same/comparable? How many shards are you actively indexing into? Are you starting indexing from Kafka once Kafka has been filled up with the documents?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 9, 2020, 3:29pm UTC](https://discuss.elastic.co/t/increase-insert-and-search-rate-in-elastic-search/254963/5 "2020-12-09T15:29:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.