Elasticsearch Output Is Failing on action => "update" due to missing document?

Elastic Stack Logstash
1

I have the following input and output blocks:

input {
  elasticsearch {
    index => "filebeat-*"
    hosts => "localhost"
    query => '{ "query": { "dis_max": {  "queries": [ { "match": { "tags": "_grokparsefailure" }}, { "match": { "type": "elbaccesslog" }} ] } } }'
    tags => [ "reindexed" ]
    docinfo => true
  }
}

output {
  if "reindexed" in [tags] {
    elasticsearch {
      hosts => [ "localhost:9200" ]
      action => "update"
      sniffing => false
      index => "%{[@metadata][_index]}"
      document_id => "%{[@metadata][_id]}"
    }
  }
}

I am experiencing two different problems related to this config:

The first problem I'm having is that documents that do not match the query in the ES input (I have verified the query using the ES search API) are being tagged with "reindexed". Why is this query not working?

The second problem I'm having is that update doesn't seem to work, as I still get "document is missing" even when I've provided the _index and the _id. It looks like it's trying to use the [type] where the [_index] should be but I have no clue why.

[2016-12-01T20:23:07,360][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>404, :action=>["update", {:_id=>"AVi70lSvttAtdvjQz4Uw", :_index=>"filebeat-2016.10.09", :_type=>"elbaccesslog", :_routing=>nil, :_retry_on_conflict=>1}, 2016-10-09T03:38:55.724Z %{host} 2016-10-09T03:38:55.724810Z prod-dash-public-elb X.X.X.X:54732 10.70.11.32:8080 0.000149 0.00204 0.000021 304 304 0 0 "GET https://web.site.com:443/ HTTP/1.1" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2
], :response=>{"update"=>{"_index"=>"filebeat-2016.10.09", "_type"=>"elbaccesslog", "_id"=>"AVi70lSvttAtdvjQz4Uw", "status"=>404, "error"=>{"type"=>"document_missing_exception", "reason"=>"[elbaccesslog][AVi70lSvttAtdvjQz4Uw]: document missing", "index_uuid"=>"T9uU1Cc7Rzep5JruP5EdGg", "shard"=>"0", "index"=>"filebeat-2016.10.09"}}}}

Can anyone help with this?

2 Likes
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.