Elasticsearch throwing number of documents in the index cannot exceed 2147483519

[Ganesh2303 (Ganesh)]

HI Team,
I'm using monthly index to store all my data in elasticsearch. i have enabled DLQ to capture the unformatted log information and in that im getting capture below error.

elasticsearch
±Could not index event to Elasticsearch. status: 400, action: ["index", {:_id=>nil, :_index=>"xx-logstash-2020.07", :_type=>"doc", :_routing=>nil}, #LogStash::Event:0x1f452fb3], response: {"index"=>{"_index"=>"xx-logstash-2020.07", "_type"=>"doc", "_id"=>"_XRRP3MBt9usZDg0br2M", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"number of documents in the index cannot exceed 2147483519"}}}c éÿÿÿÿNc 2020-07-11T19:19:28.809Z

is that any limit available to index data in single index?

[Bernt_Rostad]

This is a limitation of the underlying Lucene engine which only accepts 2^31 documents in an index. So you're basically stuck, that index won't be able to store more documents.

But since one Lucene index = one Elasticsearch shard you could avoid this problem in the future by adding more primary shards to your Elasticsearch logstash indices, since each independent shard can store 2 billion documents.

Personally, I would probably change to weekly indices instead since 2+ billion documents is a lot to search through if, say, you just want to aggregate or search on the last few days or weeks of log data. I rarely do a full month of aggregations, and if I need that I could still do it by aggregating over 4 smaller weekly indices rather than 1 big monthly.

The choice is yours, either go for weekly indices or add more primary shards to the monthly index so that it can store more documents. Good luck!

[Ganesh2303 (Ganesh)]

@Bernt_Rostad Thanks for your input and we consider your point and implement those in my configuration.