Delete the first 10000 document in an index

Elastic Stack Elasticsearch
1

Hi,
I have an annoying issue.
I have to write a script, witch can do some logrotation. I have permission only for this script, I can not configure ES. The ES has a daily index. The version of ES is 7.1.1.
The idea is to run this scipt every hour, and if there is only 15-20% free space left it deletes the last index. But it can happen the disc is too small, so it can be fill in a day. In this case this script would delete the whole daily index. So I want to delete the oldest documents in the oldest index. Eg.: the oldest 10000.
And the problem I have:
The DELETE api allows me to delete an existin index. But I tried to add an _query to it, it does not work. This query does nothing, and there is no error, just the "result" : "not_found" is worrying.

curl -sSX DELETE "localhost:9200/index-2019.06.25/_doc/_query?pretty" -H 'Content-Type: application/json' -d'
{
  "from":0,
  "size":10000,
  "sort":[{"@timestamp":{"order":"asc"}}],
  "query":{    
    "match_all": {}
  }
}'

{
  "_index" : "index-2019.06.25",
  "_type" : "_doc",
  "_id" : "_query",
  "_version" : 1,
  "result" : "not_found",
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "failed" : 0
  },
  "_seq_no" : 709750,
  "_primary_term" : 1
}

And I tied the _delete_by_query tool, but it deletes all the documents in the index. Size was set to 10000 but it deleted all the 310400 documents.

curl -sSX POST "localhost:9200/index-2019.06.25/_delete_by_query?pretty" -H 'Content-Type: application/json' -d'
{
  "size":10000,
  "sort":[{"@timestamp":{"order":"asc"}}],
  "query":{    
    "match_all": {}
  }
}'

{
  "took" : 40001,
  "timed_out" : false,
  "total" : 310400,
  "deleted" : 310400,
  "batches" : 32,
  "version_conflicts" : 0,
  "noops" : 0,
  "retries" : {
    "bulk" : 0,
    "search" : 0
  },
  "throttled_millis" : 0,
  "requests_per_second" : -1.0,
  "throttled_until_millis" : 0,
  "failures" : [ ]
}

Yes I know the best thing to do is to config ES to use hourly indexes, and delete the oldest one. It would be more efficient. But my task is not that.
Can You help me, why do not work the size setting in the delete_by_query tool? Or how should I use it to work as I want.
Or is it possible to do this? Or is there any bug?
Thank You!

2

I don't see any mention of size in the documentation: https://www.elastic.co/guide/en/elasticsearch/reference/7.1/docs-delete-by-query.html

I assume it was simply ignored.

3

There was a lot of internet search, and some results was this. Maybe that was too old.
But is there any way to make this work?

4

I'd delete by query using a date range filter most likely as you said that you want to remove the "oldest" documents.

But even better I'd simply drop the old indices like:

DELETE index-2019.06.24
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.