Unable to delete documents from index when using logstash output plugin

[truesecure (John Jack)]

My logstash plugin isn't deleting the docs in the index. It appears to be ignoring the document_id it is being passed, and instead just uses the variable name (as opposed to the value).

I have the following configuration in my logstash config:

output {
elasticsearch {
hosts => "10.10.10.128:9200"
index => "my_index"
document_type => "my_doc"
document_id => "%{my_id}"
action => "delete"
}
}

And I see this in the logs:

[2017-06-29T09:19:37,727][DEBUG][logstash.pipeline ] output received {"message"=>"{"my_id":"1:4656476"}"}}

and then ...

[2017-06-29T09:19:37,734][DEBUG][org.apache.http.wire ] http-outgoing-0 >> "{"delete":{"_id":"%{my_id}","_index":"my_index","_type":"my_doc","_routing":null}}"

Notice this (from above):

"_id":"%{my_id}"

I would have expected it to have used the actual value and read something like:

"_id":"1:4656476"

Because it doesn't use the actual value, it doesn't delete anything from the index.

What am I doing wrong?

[magnusbaeck (Magnus Bäck)]

[2017-06-29T09:19:37,727][DEBUG][logstash.pipeline ] output received {"message"=>"{"my_id":"1:4656476"}"}}

This suggests that your events don't have a my_id field but that the message field contains {"my_id":"1:4656476"}, indicating that you need a json filter or a codec => json or codec => json_lines setting in your input plugin in order to deserialize the JSON payload.

[truesecure (John Jack)]

That was it! Thanks for your help.