Remove event received from Metricbeat to Logstash


I am using a setup where Metricbeat running in remote server sends data to Logstash.

Metricbeat sends tons of data to Logstash and I don't want everything to be stored by Elasticsearch.

At each interval, Logstash receives from Metricbeat as below which are in 100s of different event.

[2017-05-11T02:30:46,633][DEBUG][logstash.pipeline ] output received {"event"=>{"@timestamp"=>2017-05-11T06:20:49.399Z, "system"=>{"process"=>{"memory"=>{"rss"=>{"pct"=>0.0, "bytes"=>0}, "size"=>0, "share"=>0}, "pgid"=>0, "name"=>"scsi_eh_3", "cpu"=>{"start_time"=>"2017-02-23T18:26:49.000Z", "total"=>{"pct"=>0.0}}, "pid"=>657, "state"=>"sleeping", "fd"=>{"limit"=>{"hard"=>4096, "soft"=>1024}, "open"=>0}, "username"=>"root", "ppid"=>2}}, "beat"=>{"hostname"=>"pte401", "name"=>"pte401", "version"=>"5.3.2"}, "@version"=>"1", "host"=>"pte401", "metricset"=>{"rtt"=>218091, "module"=>"system", "name"=>"process"}, "type"=>"metricsets", "tags"=>["beats_input_raw_event"]}}

I don't want _"cpu"=>{"start_time"} to be stored. Let's take it that is of no use for me.

How do I remove it from Logstash filter, so that it is not stored in Elasticsearch? Can you please help?

I tried to use as below, but it didn't help. I see it in Kibana still there.

mutate {
remove_field => [ "system.process.memory.share", "system.process.cpu.start_time" ]

Should be...

mutate {
    remove_field => [ "[system][process][memory][share]", "[system][process][cpu][start_time]" ]
1 Like

I'd prefer not to even ship unwanted data. See metricbeat drop_fields processor to remove fields before even sending to Logstash.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.