I am trying to delete documents using a scripted query in Elastic Search through Logstash, however whenever I do I get the following error:
[2018-03-28T22:06:17,681][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>"http://MY_ES_URL/_bulk"}
I am pointing to the correct index, correct type, and the scripted query returns correctly whenever I run with a regular script tag as seen below:
GET MY_INDEX/MY_TYPE/_search
{
"query": {
"bool" : {
"must" : {
"script" : {
"script" : {
"inline": "doc[\"MY_FIELD.keyword\"][0].toLowerCase().equals(\"1234\")",
"lang": "painless"
}
}
}
}
}
}
Here is my ElasticSearch output plugin configuration:
elasticsearch {
hosts => ["MY_ES_URL"]
action => "delete"
index => "MY_INDEX"
document_type => "MY_TYPE"
script_lang => "painless"
script_type => "inline"
script => '
doc[\"MY_FIELD.keyword\"][0].toLowerCase().equals(params.event.get("MY_FIELD").toString().toLowerCase())
'
}
It seems as if it is a problem with Elastic's output trying to use the BULK API, however I do not know how to fix this without resorting to the HTTP plugin and _delete_by_query. I am using ElasticSearch 5.4 with Logstash 5.6 and the latest ElasticSearch output plugin.