_delete_by_query returns "Gateway Timeout"

Elastic Stack Elasticsearch
1

Hi,

I am sending my logs to elasticsearch (v5.6) and want to discard logs that are older than 'x' days.
I tried to run _delete_by_query to achieve so.

There is Kibana (v5.6.1) interface through which I access the data.

Here is how query looks like

GET {IndexName}/_delete_by_query?conflicts=proceed&timeout=10m
{
"query": {
"range" : {
"@timestamp" : {
"gte" : "2017-11-12T10:50:00.000Z",
"lt" : "2017-11-12T11:00:59.942Z"
}
}
}
}

Here is response

{
"statusCode": 504,
"error": "Gateway Timeout",
"message": "Client request timeout"
}

I added ?conflicts=proceed&timeout=10m to improve reliability. But it still gives timeout.

One thing to note is that the behavior is inconsistent and mostly depends on amount of data that is getting deleted.
Trying for larger date range causes timeout to occur more frequently. But sometimes it can't even delete in 5 minute range.

What am I missing here? Why parameter timeout=10m is neglected here? Any other way to enforce timeout? Can I delete based on @timestamp in any other way?

2

If you have log data and a limited retention period, why are you not using time-based indices?

1 Like
3

Thanks for your help.

I had no knowledge of time-based indices.
Turns out, they were already enabled by my team member.
I tried using delete index method and it works well.

Thanks again.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.