How to query elasticsearch and take some fields from old data

Elastic Stack Logstash
1

Hi all,

I'm new here, hope to get some help :slight_smile:

I'm working using the logstash pipeline, for instance, I have rabbitmq as input for my logs and Elastisearch as output. Fantastic!

The problem is in the elasticsearch filter I'm using: with an if statement, if I receive a specific log class I have to look inside elasticsearch for previous logs that have the same "id" as the one that just arrived. If I found the result, I need to attach all the information from the query result into the new log, like an extension of this log with more information.

The problem is I'm not understanding the good way to query elasticsearch using "query_template" and another problem is how to make this more dynamic in term of: how do I specify my key value dynamically ?

here is my query template

  {
 	"query": {
 	"bool": {
 		"must":[{
 			"match":{
 				"id": "871df49c-acad-11e6-80f5-76304dec7eb7" --- this is the value that always changes, so i can't put static value
 			}
 		}
 	]
 	}
 },
 "_source": ["@id", "started"] -- and here i want to add the information from the previous log i found into a new fileld "started" for example, in the new log. But in my case I would love to add all the fields at once.

}

Hope I clearly explained my issue and hope to get some help!

Thank you

2

sorry for the second message, but I think the problem is elasticsearch filter is not querying correctly.

I'm not using query_template anymore and I'm using simple query

if [class] == "DPAPIINTERNAL" {
    elasticsearch {
      hosts => "10.1.10.16"
      index => "dp_audit-2017.02.16"
      query => "class:DPAPI AND request.aw:%{[aw]}"
      fields => { "request.app_instance" => "svc" }
    }

with this I want: query elasticsearch at that host using the specified index and i want to search for "class": "DPAPI" (in that index) and "request" :{"aw": "same value of the DPAPIINTERNAL"} then if the query is ok, add "app_instance" which is in the query result, in my DPAPI class log.

the result is

[2017-02-16T12:56:24,808][INFO ][logstash.filters.elasticsearch] Querying elasticsearch for lookup {:params=>{:index=>"dp_audit-2017.02.16", :q=>"class:DPAPI AND request.aw:W3BQBF", :size=>1, :sort=>"@timestamp:desc"}}
[2017-02-16T12:56:24,818][INFO ][logstash.filters.elasticsearch] Querying elasticsearch for lookup {:params=>{:index=>"dp_audit-2017.02.16", :q=>"class:DPAPI AND request.aw:5NO9I3", :size=>1, :sort=>"@timestamp:desc"}}
[2017-02-16T12:56:24,826][INFO ][logstash.filters.elasticsearch] Querying elasticsearch for lookup {:params=>{:index=>"dp_audit-2017.02.16", :q=>"class:DPAPI AND request.aw:GEWD8T", :size=>1, :sort=>"@timestamp:desc"}}

that seems to be good but i don't have any new fields...

Help please!

Thank you

3

Hi all,

This is for let the elastic community know that, the elasticsearch filter for logstash has an error in the documentation.

For instance, using the query template :

 if [type] == "end" {
         elasticsearch {
            hosts => ["es-server"]
            query_template => "template.json"
         }

whitout specifiyng the fields will result in no result.

template.json:

 {
    "query": {
      "query_string": {
       "query": "type:start AND operation:%{[opid]}"
      }
    },
   "_source": ["@timestamp", "started"]
 }

the key "_source" specify all the fields you want to take from the "old" event but is not correct to specify the "new" field in the "new" log, instead, that must be specified using a parameter for the filter that is "fields"

In my case, I did successfully like the example below:

this is the query.template:

{
    "query": {
      "query_string": {
       "query": "class:DPAPI AND request.aw:%{[aw]}"
      }
    },
   "_source": ["request"]

 }

so I'm telling I want request from the result of the query. Now let's add it in our new log "ew_key"

filter{
  if [class] == "DPAPIINTERNAL" {
    elasticsearch {
      hosts => "10.1.10.16"
      index => "dp_audit-2017.02.16"
      query_template => "/home/vittorio/Documents/elastic-queries/matching-requestaw.json"
      fields => { "request" => "new_key" }
    }

  }

}

Hope this will help somebody!

Thanks,
Vittorio

9 Likes
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.