Increase insert and search rate in elastic search

Elastic Stack Elasticsearch
1

Hi,

We are using a JAVA API to insert 100k records from a kafka topic to elastic search endpoints to a particular index. The insert rate for 100k records takes upto 30 minutes. Whereas the same 100k records can be inserted in DSE (Cassandra+Solr) in 5-7minutes. Is there a way to optimize the insert/search rate using any index optimization ?

2

Have you gone through these guidelines? What is the size of your documents? How are you indexing the data? What bulk size are you using? How many concurrent clients are indexing? What is the hardware specification of the cluster?

3

Hi Christian,
These are 1kb documents. They are ingested into kafka topic via JMeter. We are trying with the increased refresh_interval also. We have tried with 3 concurrent clients (3 instances)
The hardware specification is 4 core CPU - 10 node clusters.

4

How are you getting the data from Kafka into Elasticsearch? Is the hardware used in the two environments the same/comparable? How many shards are you actively indexing into? Are you starting indexing from Kafka once Kafka has been filled up with the documents?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.