Environment filter is not working

Elastic Stack Logstash
1

here is example:

My input file :
filter {
environment {
add_metadata_from_env => {"region" => "REGION_ENV"}
add_field => ["my_env", "%{[@metadata][region]}"]
}
}

bash-3.2$ echo $REGION_ENV
amer
bash-3.2$

My output :
{
"message" => "Hello world\r",
"@version" => "1",
"@timestamp" => "2016-02-17T00:04:13.540Z",
"host" => "0:0:0:0:0:0:0:1",
"port" => 61248,
"my_env" => "%{[@metadata][region]}"
}

Anything I am missing here ?

Thanks,
Gaurav

2

I tried this as well as mentioned on other thread.

Logstash Input :
filter {
environment {
add_metadata_from_env => {"region" => "REGION_ENV"}
add_field => ["my_env", "%{[@metadata][region]}"]
}
}

output {
stdout
{ codec => rubydebug {
metadata => true
}
}
}

Logstash Output:
{
"message" => "hello world\r",
"@version" => "1",
"@timestamp" => "2016-02-17T00:42:35.846Z",
"host" => "0:0:0:0:0:0:0:1",
"port" => 61853,
"my_env" => "%{[@metadata][region]}",
"@metadata" => {
"region" => nil
}
}

3

Works for me. Are you setting the REGION_ENV variable in the same shell from which you're invoking Logstash?

$ cat test.config
input { stdin { } }
output { stdout { codec => rubydebug { metadata => true } } }
filter {
  environment {
    add_metadata_from_env => {"region" => "REGION_ENV"}
    add_field => ["my_env", "%{[@metadata][region]}"]
  }
}
$ echo 'hello' | REGION_ENV=foo /opt/logstash/bin/logstash -f test.config
Settings: Default pipeline workers: 8
Logstash startup completed
{
       "message" => "hello",
      "@version" => "1",
    "@timestamp" => "2016-02-17T16:01:09.963Z",
          "host" => "lnxolofon",
        "my_env" => "foo",
     "@metadata" => {
        "region" => "foo"
    }
}
Logstash shutdown completed
4

Thanks a lot @magnusbaek. Appreciate your quick help. I finally got it working. I was starting logstash as sudo but setting environment variable in normal shell.

anyways, How do I remove metadata fields in the parsing key value pairs ?

I get something like this :

"region" => "amer",
"metro" => "sj",
"ibx" => "sv3",
"@metadata" => {
"region" => "amer",
"metro" => "sj",
"ibx" => "sv3"

I dont want this to be part of my output :

"@metadata" => {
"region" => "amer",
"metro" => "sj",
"ibx" => "sv3"

I did this but its not working :

filter {
environment {
add_metadata_from_env => {"region" => "REGION_ENV"}
add_metadata_from_env => {"metro" => "METRO_ENV"}
add_metadata_from_env => {"ibx" => "IBX_ENV"}
add_field => ["region", "%{[@metadata][region]}"]
add_field => ["metro", "%{[@metadata][metro]}"]
add_field => ["ibx", "%{[@metadata][ibx]}"]
remove_field => ["@metadata"]
}
}

5

The contents of the @metadata field aren't included by outputs. One exception is if you use the rubydebug codec and explicitly enable inclusion of those fields as I did in my example above.

1 Like