Logstash mutate filter

Elastic Stack Logstash
1

Hello,

logstash is getting the events coming from collectd and put it in ElasticSearch.

from kibana (plugged on ElasticSearch) I see the logstash index fields : geoip.ip, geoip.location, majflt, ...

In the input-collectd.conf file I defined a filter to remove some useless fields.
This is the filter:

filter {
mutate {
remove_field => [ "[geoip][ip]", "majflt" ]
}
}

This configuration file is correct. The "majflt" field is removed as expected but I still see the "geoip.ip" field. Does someone know the explanation ?

Regards,
Raphaël

2

Works fine for me with Logstash 2.3.2:

$ cat test.config 
input { stdin { codec => json } }
output { stdout { codec => rubydebug } }
filter {
  mutate {
    remove_field => [ "[geoip][ip]", "majflt" ]
  }
}
$ echo '{"geoip": {"ip": "1.2.3.4"}, "majflt": "foo"}' | /opt/logstash/bin/logstash -f test.config 
Settings: Default pipeline workers: 8
Pipeline main started
{
         "geoip" => {},
      "@version" => "1",
    "@timestamp" => "2016-09-07T13:21:16.736Z",
          "host" => "lnxolofon"
}
Pipeline main has been shutdown
stopping pipeline {:id=>"main"}
3

I did the same test as you and it works also (with 2.3.2 version).

With the following configuration files I always see geoip.ip in Kibana :

more input-collectd.conf

input {
udp {
port => 25826
buffer_size => 1452
codec => collectd { }
}
}
filter {
mutate {
remove_field => [ "[geoip][ip]", "majflt", "threads" ]
}
}

more output-elasticsearch.conf

output {
elasticsearch {
hosts => ["localhost:9200"]
}
}
filter {
mutate {
remove_field => [ "[geoip][ip]", "majflt", "threads" ]
}
}

4

Please replace the elasticsearch output with a stdout { codec => rubydebug } output so we can see exactly what happens.

5

I did what you said I don't see any geoip in the output file.
Maybe geoip is integrated with logstash ? I see that a "geoip" filter exists...

6

I did what you said I don't see any geoip in the output file.

Please show us. Do not describe in words what you can describe with a log snippet or something real that can't be misunderstood.

7

I cannot send the output because it is classified...

8

Yes, but you can obfuscate the data by replacing the pieces that are sensitive.

9

As it is classified I just cannot export it (even if there is no sensitive data) ...