How to delete documents older than 10 days - cant use daily indices

[Jana_P]

Hello,
I'm trying to figure out how to delete documents older than 10 days.
It sounds simple (curator etc.) - but my situation is quite complicated - I have 3 nodes cluster for DEV logs from about 30 environments.
So our indexes have names like env01-access, env01-app, env-02-access...because its good for searching in Kibana (nice organized for developers)
Access logs and app logs have different mappings so it need to be in different indexes. I'm trying to handle the number of shards under 20/1gb heap/node - now there is 315 shards and 150 indices in cluster. Daily there are about 60mio new docs in Elastic.

Now I'm using cron every day, but because of the number of documents it's really bad for performance and some of the cron commands are running for really long time (8 hours).

Example of my cron

/usr/bin/curl --silent -X POST "localhost:9200/ENV01_app*/_delete_by_query?pretty" -H 'Content-Type: application/json' -d'{"query": {"range": {"@timestamp": {"lte": "now-10d" }}}}' >/dev/null 2>&1

Can someone help me to handle this? Because of number of shards I cant use daily indices, so all logs of one type are in one index (ENV01_access...) - curator can delete only whole index. Is there a easier/better way to do that?
Thanks

[elasticforme (Sachin)]

use weekly index. then you can keep 14 days data and drop older index. Just a thought.

[Jana_P]

I tried it that way but it's not what i'm looking for. If I will use weekly index, at the end of second week it will generate two (or three) times more shards because of number of ENV and it's not ideal for cluster sizing (now it's 150 indicies/300 shards - 1 shard+1 replica...). Thanks for idea, but I'm looking for better way - if there is any :).

[borna_talebi]

You could use ILM, set rollover to 10 days, and then delete the old index.
Let's say you have an env01-app-00001 index. After 10 days ILM will create env01-app-00002 and uses that for your logs. After that You could delete your old env01-app-00001 or move it to an archive node(reduce number of primary shards and replicas).

[Christian_Dahlqvist]

Be aware that the number of shards in relation to heap size is a very general recommendation aimed at avoiding ending up with too many shards. In many cases a higher number will work quite well, so I woúld consider the suggestions around using weekly indices or ILM if even if that increases the shard count a bit.

[Jana_P]

But if I will set rollover to 10 days, I need to keep minimum 20 days until next 10 days will be covered :/. I cant delete old rollover (env01-app-00001) next 10 days, because it will delete whole index, so even if I delete it 5th day, I will only have 5 days of logs in second rollover index (env01-app-0002). The problem is that I need to keep 10 days all the time :(. (and unfortunately no archive/cold node is available)

[Jana_P]

Thanks for info. Our cluster is right now quite overloaded (there are also some "ES unreachable or down"/"collector timed out when collecting data" problems) - that's the reason I'm looking for better solution than cron. I'm quite disappointed that there isn't better way to delete logs to keep exactly 10d (management decision) - in both solutions I need to keep more days.
Second problem with rollover is that we use Logstash and in es output is ENV variable

output {
  elasticsearch {
    index => "%{env}_app"
    hosts => ["xxx"]
    template_name => "app_logs"
   }
}

and there is no possibility to generate alias dynamically :(.