Drop docs that meet certain Grok pattern

Hi all,

Currently looking to drop any documents that meet a certain Grok pattern, but am not having much luck on finding anything. In this scenario, I would like do drop anything that matches the pattern of EVENT1 in the below example

My current Grok patterns are:

  "EVENT1": "%{TIMESTAMP_ISO8601}(( '%{NOTSPACE:event.action}' %{POSINT})|:) %{LOGLEVEL:log.level} %{GREEDYDATA:message}",
  "EVENT2": """Event \[%{POSINT:event.id}\] %{NOTSPACE} \[%{TIMESTAMP_ISO8601}\] \[%{JAVACLASS:log.logger}\] \[%{LOGLEVEL:log.level}\] \[%{VSPHERE_USER}\] \[(%{NOTSPACE:vsphere.log.datacenter}|)\] \[(%{POSINT}|)\] \[A ticket for (%{DATA:vm.name}) of""",
  "EVENT3": """Event \[%{POSINT:event.id}\] %{NOTSPACE} \[%{TIMESTAMP_ISO8601}\] \[%{JAVACLASS:log.logger}\] \[%{LOGLEVEL:log.level}\] \[%{VSPHERE_USER}\] \[(%{NOTSPACE:vsphere.log.datacenter}|)\] \[(%{POSINT}|)\] \[(%{DATA:vm.name}) on""",
  "EVENT4": """Event \[%{POSINT:event.id}\] %{NOTSPACE} \[%{TIMESTAMP_ISO8601}\] \[%{JAVACLASS:log.logger}\] \[%{LOGLEVEL:log.level}\] \[%{VSPHERE_USER}\] \[(%{NOTSPACE:vsphere.log.datacenter}|)\] \[(%{POSINT}|)\]""",
  "VSPHERE_USER": """((%{HOSTNAME:user.domain}\\)?%{USERNAME:user.name}|)"""

Thanks in advance for any assistance!

Hi @demonsquatch

We should probably let @Badger answer but you certainly can just add a tag when it matches and then use a drop filter conditional on that tag.

  • Separate into two groks
  • Set the tag in the first one.
  • Then conditional grok not on that on the second set of patterns
  • Then drop based on the tag
1 Like

As @stephenb says, you could use a tag, but you assuming event.action will not exist unless the grok creates it you could use that

grok { match => { "message" => "%{TIMESTAMP_ISO8601}(( '%{NOTSPACE:event.action}' %{POSINT})|:) %{LOGLEVEL:log.level} %{GREEDYDATA:message}" } }
if [event.action] { drop {} }
grok { match => { "message" => [ an array, the rest of the patterns ] } }

Also, are you using Logstash or Elasticsearch ingest pipelines?

The post is Logstash forum, but it has the tag ingest-pipeline, so it is a little confusing.

I'm asking because if you are using Logstash and want to create ECS fields, as it seems to be the goal, you need to reference the fields using [event][action], [user][name], [log][level] etc.

In your grok you are using event.action, user.name etc, on Logstash those are not the same thing.

For example, using user.name in the following pattern %{USERNAME:user.name} will create a field with a literal dot in the name.

    "user.name": "username"

The ECS field is a object named user with a nested field named name:

    "user": {
        "name": "username"

To create a field like this in Logstash you need to use %{USERNAME:[user][name]} in this example.

1 Like

Thank you for the response and pointing that out - I am doing it with Elasticsearch ingest pipeline and have moved it to the correct forum

Worked beautifully, thank you so much!

You are not the only one that is/was confused. A colleague and I spent at least 5 minutes in a discussion with an Elastic sales representative about pipelines before it dawned on us that she was talking about Elasticsearch Ingest Pipelines and we were inquirering about Logstash Pipelines and why and when it would be advantageous to have more than one.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.