Unable to delete documents from index when using logstash output plugin

My logstash plugin isn't deleting the docs in the index. It appears to be ignoring the document_id it is being passed, and instead just uses the variable name (as opposed to the value).

I have the following configuration in my logstash config:

output {
elasticsearch {
hosts => "10.10.10.128:9200"
index => "my_index"
document_type => "my_doc"
document_id => "%{my_id}"
action => "delete"
}
}

And I see this in the logs:

[2017-06-29T09:19:37,727][DEBUG][logstash.pipeline ] output received {"message"=>"{"my_id":"1:4656476"}"}}

and then ...

[2017-06-29T09:19:37,734][DEBUG][org.apache.http.wire ] http-outgoing-0 >> "{"delete":{"_id":"%{my_id}","_index":"my_index","_type":"my_doc","_routing":null}}"

Notice this (from above):

"_id":"%{my_id}"

I would have expected it to have used the actual value and read something like:

"_id":"1:4656476"

Because it doesn't use the actual value, it doesn't delete anything from the index.

What am I doing wrong?

[2017-06-29T09:19:37,727][DEBUG][logstash.pipeline ] output received {"message"=>"{"my_id":"1:4656476"}"}}

This suggests that your events don't have a my_id field but that the message field contains {"my_id":"1:4656476"}, indicating that you need a json filter or a codec => json or codec => json_lines setting in your input plugin in order to deserialize the JSON payload.

That was it! Thanks for your help.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.