Mutate filter remove_field does not work with some field names

Elastic Stack Logstash
1

Hi,
I am trying to remove some fields from a json message.
I am trying this with mutate like this :

input {
  file {
    type => "log-mongo"
    codec => "json"
    path => "first_logstash.log"
  }
  file {
    type => "log-kafka"
    codec => "json"
    path => "/tmp/tmp.json"
  }
}

filter {
  if [type] == "log-kafka" {
    mutate {
      remove_field => ["@version","@timestamp","path","host","type" ]
    }
  }
}

output {
  if [type] == "log-mongo" {
    mongodb {
      codec => "json"
      collection => "logs"
      database => "mydb"
      uri => "mongodb://127.0.0.1"
      isodate => true
    }
  }

  if [type] == "log-kafka" {
    file {
      path => "/tmp/o"
      codec => rubydebug
    }
  }
}

If i use the above config, this actually breaking the output in general. No messages will appear on the output file. No exception, no nothing.
If i just remove the "type" from the "remove_field", it wokrs.

this works:
mutate { remove_field => ["@version","@timestamp","path","host"] }

I want to remove this "type" field.

Any ideas how to achieve this ?

It seems that logstash has some "reserved words" :frowning:

2

You may need to apply a custom ruby filter to remove the type field.

3

Thank you for the answer.

and .. that's too bad, as ruby is not really my strength

4

I should have given more context. I apologize.

I also should have linked to the ruby filter: https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html

Here's an example that may help you:

cat ~/logstash-test.conf
input { stdin {} }

filter {
  mutate { add_field => { "foo" => "bar" }}
  ruby { code => "event.remove('foo')" }
}

output { stdout { codec => rubydebug } }

This demonstrates the ruby code to remove the field foo from the event.

The output of this test:

bin/logstash -f ~/logstash-test.conf --verbose
starting agent {:level=>:info}
starting pipeline {:id=>"main", :level=>:info}
Settings: Default pipeline workers: 8
Starting pipeline {:id=>"main", :pipeline_workers=>8, :batch_size=>125, :batch_delay=>5, :max_inflight=>1000, :level=>:info}
Pipeline main started
This is my message
{
       "message" => "This is my message",
      "@version" => "1",
    "@timestamp" => "2016-06-29T16:11:44.282Z",
          "host" => "macbuntu2"
}
^CSIGINT received. Shutting down the agent. {:level=>:warn}
stopping pipeline {:id=>"main"}
Closing inputs {:level=>:info}
Closed inputs {:level=>:info}

Note that no field named foo made it to the output.

For you, the last filter should probably be

ruby {
  code => "event.remove('type')"
}
5

Thank you again for your answer.
As usual the problem was me.

What actually happened:

When i remove the type in the filer, the next IF in the output section fails, as there is no such field "type" anymore.
So mutate works as expected.

So when one wants to get rid of the type, but still be able to compare against this field in another if:

add_field => { "[@metadata][type]" => "log-kafka" }

and then

if [@metadata][type] == "log-kafka" {}

Hope my mistake will save some time for somebody else :slight_smile:

2 Likes