Logstash does not execute certain queries correctly

[Stefan_Sabolowitsch]

Hi there
i do not understand the behavior of logstash.
Although the field is_read exists, a successful query is still performed and an e-mail is sent.

    input {
      elasticsearch {
        hosts => "https://elastic01:9200"
        ssl => true
        ca_file => "/etc/logstash/certs/http_ca.crt"
        user => "elastic"
        password => "password"
        index => "test-index-default"
        query => {"query":{"bool":{"must_not":[{"exists":{"field":"is_read"}}],"must":[{"range":{"@timestamp":{"gte":"now-1h"}}}]}}}
        schedule => "* * * * *"
        size => 500
        scroll => "1m"
        docinfo => true
        docinfo_target => "[@metadata][doc]"
      }
    }
    filter {
      json {
        source => "message"
      }
      mutate {
        replace => {"is_read"=> "true"}
      }
    }
    output {
      stdout {}
      email {
        to => "testuser@group.com"
        address => "test.smtp"
        subject => "Filebeat"
        body => 'Test Message'
        port => "25"
        #username => "xxx@gmail.com"
        #password => "****"
        #use_tls => true
       }
      elasticsearch {
        hosts => "https://elastic01:9200"
        ssl => true
        cacert => "/etc/logstash/certs/http_ca.crt"
        user => "elastic"
        password => "password"

        retry_on_conflict => 5
        index => "test-index-default"
        document_type => "%{[@metadata][doc][_type]}"
        document_id => "%{[@metadata][doc][_id]}"
        action => "update"
      }
    }

The query itself can be carried out successfully in Kibana.
Hence the question, why is this query successful?
{"query":{"bool":{"must_not":[{"exists":{"field":"is_read"}}],"must":[{"range":{"@timestamp":{"gte":"now-1h"}}}]}}}
Although the field exists and therefore a mail is still sent every minute.

thanx for any help here
Stefan

[Wave (Andrew Sullivan)]

Hi @Stefan_Sabolowitsch,

Going to try my best to answer your questions by describing what I see happening. Hopefully my description helps you figure out what might need to be changed.

1. Input: Runs every minute and grabbing documents where is_read field doesn't exist.
2. Filter: Message field is converted to json. is_read is set to true
3. Output: The document goes to standard out. The same document goes to email. The same document goes back to elasticsearch.

I'm not sure exactly the behavior you are going for, but as you can see each output module is independent of each other. Depending upon what you want to you can place conditionals around them.

Finally, you don't mention which version you are running but document_type for elasticsearch output is deprecated.