Elastic filter not executing query? Bool etc. supported?

[Sjaak01 (Sjaak)]

Hi,

I'm trying to use the elastic filter to get some data from a index. However the elastic filter doesn't appear to be doing anything. When starting logstash I see New ElasticSearch filter client {:hosts=>["https://.url.found.io:9243"]} but the queried data doesn't appear. There aren't any errors in the logs either.

I had similar issues with the elastic input before, where the Docs claim elastic DSL is fully supported but in reality only very basic queries work and anything including bool filters etc. doesn't work.

config

filter {
      elasticsearch {
         hosts => ["https://url.found.io:9243"]
         password => "pass"
         user => "user"
         query_template => "template.json"
         fields => { "hdg" => "HEADING" }
      }
}

output {
  stdout {
   codec => rubydebug
 }
}

template (confirmed this gives the correct results in the Kibana console)

{
    "query": {
    "bool" : {
      "filter": [
        {
              "range" : {
                "@timestamp" : {
                "lt": "%{[@timestamp]}"
                 }
               }
              }
              ],
      "must" : {
        "term" : { "ID" : "%{[ID]}" }
      },
            "should": [{
                "exists": {
                    "field": "hdg"
                }
            },
            {
                "exists": {
                    "field": "hdg2"
                }
            }
            ],
            "minimum_should_match" : 1
    }
  },
  "size": 1,
  "_source": {
    "includes": [
      "@timestamp",
      "name",
      "hdg",
      "ID"]
  },
  "sort": [
    {
      "@timestamp": {
        "order": "desc"
      }
    }
  ]
}

The original document I'm ingesting into Logstash contains the ID and @timestamp fields. How can I further troubleshoot this issue?

[Sjaak01 (Sjaak)]

Query works when using basic queries. however I need to have at least a range filter in there. In KQL you can do @timestamp < but of course this doesn't work in the filter either.

template

{
  "size": 1,
  "sort" : [ { "@timestamp" : "desc" } ],
  "query": {
    "query_string": {
      "query": "IMO:%{[ID]} AND hdg:*"
    }
  },
  "_source": ["@timestamp", "ID", "hdg"]
}

Why isn't this clearly mentioned in the Docs?

[Sjaak01 (Sjaak)]

For those who come across this issue in the future, @timestamp:{* TO %{[@timestamp]}} can used to set a date range.