How to access the value in the logstash metadata


(BIJAY) #1

Hi,

I want to access the value of the dictionary in the metadata dynamically. How can I do that in logstash filter section?

e.g.

    if [@metadata][key] == "test" {
        <some action>
    }

Thanks
Bijay


(Christian Dahlqvist) #2

Something like this?

input {
  generator {
    lines => ['message1']
    count => 1
    add_field => { "[@metadata][key]" => "test" }
  }
}

filter {
  if [@metadata][key] == "test" {
    mutate {
      add_tag => ["found"]
    }
  }
}

output {
  stdout { codec => rubydebug { metadata => true }}
}

(BIJAY) #3

Thanks for the reply Christian.

I want to explain little bit more on that.

This is my input logline (for testing only I have used stdin and stdout) and I want to get the value of the first key dynamically.

[severity_type][severity] can be anything, e.g. [severity_type][info] or [severity_type][debug] or [severity_type][warning]

input {
    stdin { codec => json }
}
filter {
    
#    if [severity_type][info] == "drop" {
#        drop { }
#    }

#    if [severity_type][**KEY**] == "drop" {
#        drop { }
#    }

}
output {
    stdout { codec => rubydebug
      { metadata => true }
    }
}

Input loglines

{"nodetype":"haproxy","ssinst":"001","component":"kafka","severity":"info","logsource":"log"}

{"nodetype":"haproxy","ssinst":"001","component":"logstash","severity":"info","logsource":"log"}

{"nodetype":"haproxy","ssinst":"001","component":"logstash","logsource":"log"}

{"nodetype":"haproxy","ssinst":"001","logsource":"log"}


(BIJAY) #4

Works fine when I have the config like this, but I want to get the field value of the severity_type dynamically. I meant first field value.

input {
    stdin { codec => json }
}
filter {
    
    translate {
        field => "component"
        destination => "logstat"
        dictionary_path => "/tmp/logdrop.yml"
    }
    json {
        source => "logstat"
        target => "severity_type"
    }

    if [severity_type][info] == "drop" {
        drop { }
    }

}
output {
    stdout { codec => rubydebug
      { metadata => true }
    }
}

(Christian Dahlqvist) #5

I do not understand your example. can you please clarify? The field name definition can as far as I know not be dynamic.


(BIJAY) #6

Ohh, ok. Is there any work around to get the field name definition dynamic? Like in an array it can be done sometime like [arr][0] or [arr.0]


(Christian Dahlqvist) #7

I suspect you may need to use a ruby filter for that.


(BIJAY) #8

I don't much knowledge about the ruby code. Would you please help me?


(Christian Dahlqvist) #9

I still do not understand your example. If possible I would probably recommend restructuring your data instead to avoid having to do this. The Ruby filter is unfortunately not something I have used to any great extent, so will not be able to help you there...


(BIJAY) #10

I want to drop the logs based on the severity type by updating the dictionary path.

input {
    stdin { codec => json }
}
filter {
    
    translate {
        field => "component"
        destination => "logstat"
        dictionary_path => "/tmp/logdrop.yml"
    }
    json {
        source => "logstat"
        target => "severity_type"
    }

    if [severity_type][info] == "drop" {
        drop { }
    }

}
output {
    stdout { codec => rubydebug
      { metadata => true }
    }
}

This is the /tmp/logdrop.yml

sh-4.2# cat /tmp/logdrop.yml
---
logstash: '{"info":"drop"}

(BIJAY) #11

I have another open thread related to this.


(BIJAY) #12

Any suggestions on this Christian?


(Christian Dahlqvist) #13

Please do not open multiple threads for the same problem. I still do not understand what you are trying to do and why you need dynamic field names, so do not have any suggestions at this time.


(BIJAY) #14

Both are not exactly the same issue though they are related. In the other thread I wanted to get some suggestion on how to proceed and in this thread I have sought help on the generic thing, i.e. getting dynamic field names.


(BIJAY) #15

Is it possible to get the value based on the position of the parameter? Something similar as below (this doesn't work):

if [severity_type.index][0] == "drop" {
        drop { }
    }