How to access the value in the logstash metadata

Elastic Stack Logstash
1

Hi,

I want to access the value of the dictionary in the metadata dynamically. How can I do that in logstash filter section?

e.g.

    if [@metadata][key] == "test" {
        <some action>
    }

Thanks
Bijay

2

Something like this?

input {
  generator {
    lines => ['message1']
    count => 1
    add_field => { "[@metadata][key]" => "test" }
  }
}

filter {
  if [@metadata][key] == "test" {
    mutate {
      add_tag => ["found"]
    }
  }
}

output {
  stdout { codec => rubydebug { metadata => true }}
}
3

Thanks for the reply Christian.

I want to explain little bit more on that.

This is my input logline (for testing only I have used stdin and stdout) and I want to get the value of the first key dynamically.

[severity_type][severity] can be anything, e.g. [severity_type][info] or [severity_type][debug] or [severity_type][warning]

input {
    stdin { codec => json }
}
filter {
    
#    if [severity_type][info] == "drop" {
#        drop { }
#    }

#    if [severity_type][**KEY**] == "drop" {
#        drop { }
#    }

}
output {
    stdout { codec => rubydebug
      { metadata => true }
    }
}

Input loglines

{"nodetype":"haproxy","ssinst":"001","component":"kafka","severity":"info","logsource":"log"}

{"nodetype":"haproxy","ssinst":"001","component":"logstash","severity":"info","logsource":"log"}

{"nodetype":"haproxy","ssinst":"001","component":"logstash","logsource":"log"}

{"nodetype":"haproxy","ssinst":"001","logsource":"log"}

4

Works fine when I have the config like this, but I want to get the field value of the severity_type dynamically. I meant first field value.

input {
    stdin { codec => json }
}
filter {
    
    translate {
        field => "component"
        destination => "logstat"
        dictionary_path => "/tmp/logdrop.yml"
    }
    json {
        source => "logstat"
        target => "severity_type"
    }

    if [severity_type][info] == "drop" {
        drop { }
    }

}
output {
    stdout { codec => rubydebug
      { metadata => true }
    }
}
5

I do not understand your example. can you please clarify? The field name definition can as far as I know not be dynamic.

6

Ohh, ok. Is there any work around to get the field name definition dynamic? Like in an array it can be done sometime like [arr][0] or [arr.0]

7

I suspect you may need to use a ruby filter for that.

8

I don't much knowledge about the ruby code. Would you please help me?

9

I still do not understand your example. If possible I would probably recommend restructuring your data instead to avoid having to do this. The Ruby filter is unfortunately not something I have used to any great extent, so will not be able to help you there...

10

I want to drop the logs based on the severity type by updating the dictionary path.

input {
    stdin { codec => json }
}
filter {
    
    translate {
        field => "component"
        destination => "logstat"
        dictionary_path => "/tmp/logdrop.yml"
    }
    json {
        source => "logstat"
        target => "severity_type"
    }

    if [severity_type][info] == "drop" {
        drop { }
    }

}
output {
    stdout { codec => rubydebug
      { metadata => true }
    }
}

This is the /tmp/logdrop.yml

sh-4.2# cat /tmp/logdrop.yml
---
logstash: '{"info":"drop"}
11

I have another open thread related to this.

12

Any suggestions on this Christian?

13

Please do not open multiple threads for the same problem. I still do not understand what you are trying to do and why you need dynamic field names, so do not have any suggestions at this time.

14

Both are not exactly the same issue though they are related. In the other thread I wanted to get some suggestion on how to proceed and in this thread I have sought help on the generic thing, i.e. getting dynamic field names.

15

Is it possible to get the value based on the position of the parameter? Something similar as below (this doesn't work):

if [severity_type.index][0] == "drop" {
        drop { }
    }
16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.