Safely remove tcp input plugin's "host" and "port" fields

Elastic Stack Logstash
1

The tcp input plugin adds host and port fields to received events.

Ideally we don't want to add these but since this isn't documented, there doesn't seem to be a way to remove them.

What is the best filter to safely remove them? For example, we can't blindly:

filter {
  mutate {
    remove_field => [
      "host",
      "port"
    ]
  }
}

since some of our received events are using ECS and this will destroy anything under host (e.g. host.hostname)

If there a way to test in logstash for this? Or should we use a ruby filter?

EDIT:

I would like to do something like:

filter {
   ruby {
      code => "event.delete('host') if String === event.get('host')"
   }  
}

but the ruby API only provides get and set; you can't delete a field.

EDIT:

unless I hear "there's a better way to do this" here's what I'm settling on:

filter {
  ruby {
    code => "event.set('host', {}) if String === event.get('host')" # set by input plugins, we don't care
    id => "remove_input_noise"
    remove_field => [
      "port" # set by input plugins, we don't care
    ]
  }
}

if elastic receives the blank hash under host, it won't index anything so there's no extra noise in the event. It's still in the original document though.

2

Not so, there is an event.remove method that deletes a field.

3

Is it unsupported? It's not in the ruby API documentation.

4

They may be undocumented but event.remove or event.cancel are both used a lot.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.