Discarding unnecessary data

Elastic Stack Logstash
1

Hello,

I am just starting with logstash and writing filters, and trying to better understand how it works. We want to push all syslog messages to logstash but I want to ensure that events are only sent to elastic if they match a grok filter. So if (using the Cisco example) I have a grok {
match => [
# IOS
"message", "%{SYSLOG5424PRI}(%{NUMBER:log_sequence#})? .... etc

What happens to messages that do not match the Grok pattern specified - are they discarded?

Also, what if I want to discard a portion of a message. I know that %{NUMBER:log_sequence#} would assign the number to "log_sequence), What if I want to drop the number altogether?

2

Hi there,

to answer your questions:

What happens to messages that do not match the Grok pattern specified - are they discarded?

Not by default. By default they are ingested together with a field tags with value _grokparsefailure in it.

We want to push all syslog messages to logstash but I want to ensure that events are only sent to elastic if they match a grok filter

If the event matches a grok, as you said, given fields will be populated (for example that log_sequence). Then, you can either drop the event in the filter section if a given field is not popuated or ingest the doc only if that field is populated. So:

filter {
  ...
  if ![log_sequence] {
    drop{}
  }
}

or

output {
  if [log_sequence] {
    ..whatever output you want..
  }
}

Choose the solution that better fits your needs.

Also, what if I want to discard a portion of a message. I know that %{NUMBER:log_sequence#} would assign the number to "log_sequence), What if I want to drop the number altogether?

You can simply not assign anything to that part. In that case leaving %{NUMBER} without the log_sequence part will cause that part to go lost and not stored in your final document.

3

Got it - Thank you

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.