[Logstash] How to drop message if field is not a number

Elastic Stack Logstash
1

Hi Logstash gurus,

I need to drop the messages that contain specific fields that are not a number.

The filter I have is:

filter {
  csv {
      separator => ","
      skip_header => "true"
      columns => ["process-name","upload-bw","download-bw","process-owner","filename","hostname"]
	  convert => {
		"upload-bw" => "float"
		"download-bw" => "float"
	  }
  }
}

Sometimes, upload-bw and download-bw contain characters, so I get conflicts in the index pattern and the dashboards don't render.

So I want to drop all those messages with upload-bw and download-bw that are not a float.

Can you help?

Thank you,
Catalin

2

You could try

    ruby {
        code => '
            [ "upload-bw", "download-bw" ].each { |x|
                if ! event.get(x).is_a? Float; event.remove(x); end
            }
        '
    }
1 Like
3

Or you could use a grok filter in stead of csv to parse your lines, if the upload-bw and download-bw do not contain numbers then you have no match and can drop the event based on the _grokparsefailure tag.

filter {
    grok {
        match => { "message" => "%{WORD:process-name}, %{NUMBER:upload-bw}, %{NUMBER:download-bw}, (?<process-owner>([a-zA-Z]*)), (?<filename>([a-zA-Z\-\_\.]*)), %{WORD:hostname}" }
    }
    if "_grokparsefailure" in [tags] {
        drop { }
    }
}

This is just an example, you would have to spend time debugging your grok/regex to match your data, while your csv solution with the ruby code works out of the box.
I'm adding this comment as an option because it might be worth to investage what solution is more "expensive" in resources.

4

Hi Badger, Anton, thank you!
Let me try to test both approaches and will get back to you...

Catalin

5

Hi Badger,

I tried the ruby code you suggested and it works! Well, it's not dropping the whole message, but it empties the upload-bw and download-bw fields so the index doesn't conflict anymore.

I changed it a bit, instead of remove I put event.set(x, 0.0).

Thank you!
Catalin

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.