Delete document in ElasticSearch Output Plugin using Script

I am trying to delete documents using a scripted query in Elastic Search through Logstash, however whenever I do I get the following error:

[2018-03-28T22:06:17,681][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://MY_ES_URL/_bulk"}

I am pointing to the correct index, correct type, and the scripted query returns correctly whenever I run with a regular script tag as seen below:

GET MY_INDEX/MY_TYPE/_search
{
    "query": {
        "bool" : {
            "must" : {
                "script" : {
                    "script" : {
                        "inline": "doc[\"MY_FIELD.keyword\"][0].toLowerCase().equals(\"1234\")",
                        "lang": "painless"
                     }
                }
            }
        }
    }
}

Here is my ElasticSearch output plugin configuration:

   elasticsearch {
        hosts => ["MY_ES_URL"]
        action => "delete"
        index => "MY_INDEX"
        document_type => "MY_TYPE"
        script_lang => "painless"
        script_type => "inline"
        script => '
            doc[\"MY_FIELD.keyword\"][0].toLowerCase().equals(params.event.get("MY_FIELD").toString().toLowerCase())
        ' 
    }

It seems as if it is a problem with Elastic's output trying to use the BULK API, however I do not know how to fix this without resorting to the HTTP plugin and _delete_by_query. I am using ElasticSearch 5.4 with Logstash 5.6 and the latest ElasticSearch output plugin.

I tried to replicate what the output plugin is doing through the BULK API and I ran the following Query:

POST _bulk
{
  "delete": {
    "_index": "MY_INDEX",
    "_type": "MY_TYPE",
    "query": {
      "bool": {
        "must": {
          "script": {
            "script": {
              "inline": "doc[\"MY_FIELD.keyword\"][0].toLowerCase().equals(\"1234\")",
              "lang": "painless"
            }
          }
        }
      }
    }
  }}

The error I am getting is this:

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "Malformed action/metadata line [1], expected a simple value for field [query] but found [START_OBJECT]"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "Malformed action/metadata line [1], expected a simple value for field [query] but found [START_OBJECT]"
  },
  "status": 400
}

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.