I have a cluster configured in Kubernetes by Elastic Operator. all components of version 7.1. I'm sending logs from Kafka topic to Elastic with KafkaConnect
And I faced with strange behavior. When I started KafkaConnector and point to the non-existing index the indexing rate is pretty good (~4k docs/s). But if then stop connector and start it again, or just point connector to the existing index the rate is much lower (about 10x times).
Here is a graph:
the red mark when the connector was started and initiated a new index creation. Then I stopped the connector and started it again and got the rate marked in blue
I have not configured any additional configuration for the cluster and index except:
"refresh_interval" : "60s"
So, I'm trying to understand why the indexing rate for the existing index is so low and is there any way to improve it
Are you sending exactly the same records that already exist in the index?
Elasticsearch doesn't keep any state between requests. So, the first record creates an index but then there is really no difference. So, My gut feel here that the problem is either in the way you index records in the first and the second iterations, or something happened at 10:27 that affected the index or the records that you are indexing second time are very different or exactly the same as I mentioned above.
I'm sending application logs from the Kafka topic. And this behavior is reproducible, so I can point Kafka Connect to the new (non-existing) index and will have a good performance (~4k doc/s). Then I can stop Kafka Connect and start it again and it will be able to index only 300-500 docs/s. I did this couple of times.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
).