HI Team,
I'm using monthly index to store all my data in elasticsearch. i have enabled DLQ to capture the unformatted log information and in that im getting capture below error.
elasticsearch ±Could not index event to Elasticsearch. status: 400, action: ["index", {:_id=>nil, :_index=>"xx-logstash-2020.07", :_type=>"doc", :_routing=>nil}, #LogStash::Event:0x1f452fb3], response: {"index"=>{"_index"=>"xx-logstash-2020.07", "_type"=>"doc", "_id"=>"_XRRP3MBt9usZDg0br2M", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"number of documents in the index cannot exceed 2147483519"}}}c éÿÿÿÿNc 2020-07-11T19:19:28.809Z
is that any limit available to index data in single index?
This is a limitation of the underlying Lucene engine which only accepts 2^31 documents in an index. So you're basically stuck, that index won't be able to store more documents.
But since one Lucene index = one Elasticsearch shard you could avoid this problem in the future by adding more primary shards to your Elasticsearch logstash indices, since each independent shard can store 2 billion documents.
Personally, I would probably change to weekly indices instead since 2+ billion documents is a lot to search through if, say, you just want to aggregate or search on the last few days or weeks of log data. I rarely do a full month of aggregations, and if I need that I could still do it by aggregating over 4 smaller weekly indices rather than 1 big monthly.
The choice is yours, either go for weekly indices or add more primary shards to the monthly index so that it can store more documents. Good luck!
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.