Elasticsearch indexing rate fluctuation

[elk_user1]

Hi there,

I am using kafka connect to load data from kafka into elasticsearch index (custom mapping using dynamic template). I am monitoring performance via X-Pack and noticed that indexing rate fluctuates a lot every 10 minutes. Is there anything which I am missing which can be set to get stable indexing rate?

Thanks in advance.

[polyfractal (Zachary Tong)]

That's a very complicated question

The first thing to do is identify the bottleneck. Is it kafka being slow providing documents? Is the queue emptying out? Does the ES server share resources with any other process that can kick on (I've seen background cron jobs periodically destroy performance)? If it's elasticsearch, what resources are bottlenecked? CPU/disk/etc?

Once you identify the location, you can start to look into tweaking/tuning. But until then it's pretty much impossible to advise, since there could be dozens of reasons why you are seeing that graph

[elk_user1]

Thanks for replying. We have a dedicated ES cluster with 6 nodes, so there are no other processes running. And CPU and memory are fine.

I tried to set refresh interval to 30 seconds as well as -1, but I don't see any significant performance improvement. I guessed setting it would impact the indexing rate. Is this expected? Any help appreciated.

Thanks.