Currently looking to drop any documents that meet a certain Grok pattern, but am not having much luck on finding anything. In this scenario, I would like do drop anything that matches the pattern of EVENT1 in the below example
As @stephenb says, you could use a tag, but you assuming event.action will not exist unless the grok creates it you could use that
grok { match => { "message" => "%{TIMESTAMP_ISO8601}(( '%{NOTSPACE:event.action}' %{POSINT})|:) %{LOGLEVEL:log.level} %{GREEDYDATA:message}" } }
if [event.action] { drop {} }
grok { match => { "message" => [ an array, the rest of the patterns ] } }
Also, are you using Logstash or Elasticsearch ingest pipelines?
The post is Logstash forum, but it has the tag ingest-pipeline, so it is a little confusing.
I'm asking because if you are using Logstash and want to create ECS fields, as it seems to be the goal, you need to reference the fields using [event][action], [user][name], [log][level] etc.
In your grok you are using event.action, user.name etc, on Logstash those are not the same thing.
For example, using user.name in the following pattern %{USERNAME:user.name} will create a field with a literal dot in the name.
{
"user.name": "username"
}
The ECS field is a object named user with a nested field named name:
{
"user": {
"name": "username"
}
}
To create a field like this in Logstash you need to use %{USERNAME:[user][name]} in this example.
You are not the only one that is/was confused. A colleague and I spent at least 5 minutes in a discussion with an Elastic sales representative about pipelines before it dawned on us that she was talking about Elasticsearch Ingest Pipelines and we were inquirering about Logstash Pipelines and why and when it would be advantageous to have more than one.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.