Data loss ,bulk rejected high in elastic search

I am sending data from filebeat (6.3) from 138 servers to elastic search 6.3.( 3 master and 17 data nodes with (6data node 30gb memory and 11 with 15gb memeory))

getting bulk failure in filebeat when checking in elastic search log its bulk rejected.

what should be the queue size for production ? currently using write thread popl with 2000 queue size.

getting data loss

If you are using Filebeat, the fact that you are seeing bulk rejections should not lead to data loss as Filebeat will retry.

Have a look at this blog post around the reason behind bulk rejections. How many indices and shards are you actively indexing into? Are you indexing into all data nodes or is the different node sizes due to the use of a hot-warm architecture? If you are indexing into a large number of indices/shards, you probably want to have a look at this blog post as well.

Have you followed these guidelines to make sure Elasticsearch is indexing efficiently as possible?

Everything Christian said it's spot on. Rejections are generally not a data loss problem. I would look at elastic search thread pool end point to see which node is giving you the rejections. I have experience with indexing into mixed hardware clusters and the lower specced nodes rejecting requests. I expect this is your issue. For heavy indexing try to use your best nodes - using index allocations. Then move to your lower specced nodes.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.