Unable to delete the documents when the index is full

Elastic Stack Elasticsearch
#1

I am unable to delete the documents from an index having the maximum number of documents (2147483519).

getting the below exception. Please help me to get this resolved.

"cause" : {
        "type" : "illegal_argument_exception",
        "reason" : "number of documents in the index cannot exceed 2147483519"
      }

below is the query

POST logs-68-2425/_delete_by_query
{
  "query": {
    "range": {
      "@timestamp": {
        "lt": 1602374400000,
        "format": "epoch_millis"
      }
    }
  }
}
#2

You need to reindex the data instead IMO.

#3

Have you tried using the split index API (assuming you are on a recent version)? Not sure if it would work on a completely full shard but it could be worth exploring...

#4

That was my initial answer to be honest but the error message makes me think that the limit is per index and not per shard? Unless this is an error message coming directly from Lucene?

#5

I assumed it was a single shard index given that the default now is 1 primary shard per index.

1 Like
#6

As you can see from the discussion @subash it would be useful to know which version of Elasticsearch you are using as well as get some index and/or shard statistics.

#7

Yeah it's from Lucene so "index" means "shard" in Elasticsearch terminology.

The fundamental issue is that deleting a document requires indexing another document to record the delete, and that happens before the deleted document goes away. I suspect that splitting the index would work, but if that doesn't work then reindexing would work too.

1 Like