How to query elasticsearch and take some fields from old data

Elastic Stack Logstash
3

Hi all,

This is for let the elastic community know that, the elasticsearch filter for logstash has an error in the documentation.

For instance, using the query template :

 if [type] == "end" {
         elasticsearch {
            hosts => ["es-server"]
            query_template => "template.json"
         }

whitout specifiyng the fields will result in no result.

template.json:

 {
    "query": {
      "query_string": {
       "query": "type:start AND operation:%{[opid]}"
      }
    },
   "_source": ["@timestamp", "started"]
 }

the key "_source" specify all the fields you want to take from the "old" event but is not correct to specify the "new" field in the "new" log, instead, that must be specified using a parameter for the filter that is "fields"

In my case, I did successfully like the example below:

this is the query.template:

{
    "query": {
      "query_string": {
       "query": "class:DPAPI AND request.aw:%{[aw]}"
      }
    },
   "_source": ["request"]

 }

so I'm telling I want request from the result of the query. Now let's add it in our new log "ew_key"

filter{
  if [class] == "DPAPIINTERNAL" {
    elasticsearch {
      hosts => "10.1.10.16"
      index => "dp_audit-2017.02.16"
      query_template => "/home/vittorio/Documents/elastic-queries/matching-requestaw.json"
      fields => { "request" => "new_key" }
    }

  }

}

Hope this will help somebody!

Thanks,
Vittorio

9 Likes