How to drop events only from specific module

Elastic Stack Beats
1

Hello,

I want to drop all events that filebeat was unable to parse.

I tried a few things my last attempt looks like this.

processors:
  - drop_event:
         when:
         and:
         - equals.event.module: cisco
         - equals.message: "failed to find message"

But I can still see the events with the message "failed to find message" in kibana log stream.
How can I make it work?

Filebeat version is 7.9.2

2

Is that yaml correctly indented? Can you try playing with the indentations of the yamls? It's a common source of issues

3

The config is loading and starting. When I had a tab in ymal there was an issue with the starting of filebeat.

the weird thing is when I do something like this

POST filebeat-7.9.2-cisco-2020.11.03/_search
{
   "size": 1,
   "sort": { "@timestamp": "desc"},
   "query": {
      "match": {"message":"failed to find message"}
   }
}

This is a document I get back.

{
  "took" : 1202,
  "timed_out" : false,
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 8374,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [
      {
        "_index" : "filebeat-7.9.2-cisco-2020.11.03",
        "_type" : "_doc",
        "_id" : "EYtVkHUBMI-dbCtF8X18",
        "_score" : null,
        "_source" : {
          "agent" : {
            "hostname" : "name",
            "name" : "name",
            "id" : "4b069842-f4a1-4420-9a89-7e093d2cba6f",
            "ephemeral_id" : "0285f0b0-01e2-4691-9171-18e5a781cae2",
            "type" : "filebeat",
            "version" : "7.9.2"
          },
          "log" : {
            "file" : {
              "path" : "/var/log/syslog/name.log"
            },
            "original" : "%ASA-6-302014: Teardown TCP connection 23913857 for name-1:ip-2/49903 to name-2:ip-3/23 duration 25:54:08 bytes 4095435 TCP FINs",
            "offset" : 2207,
            "level" : "informational"
          },
          "message" : "Teardown TCP connection 23913857 for name-1:ip-2/49903 to name-2:ip-3/23 duration 25:54:08 bytes 4095435 TCP FINs",
          "fileset" : {
            "name" : "asa"
          },
          "error" : {
            "message" : [
              "Provided Grok expressions do not match field value: [Teardown TCP connection 23913857 for name-1:ip-2/49903 to name-2:ip-3/23 duration 25:54:08 bytes 4095435 TCP FINs]"
            ]
          },
          "tags" : [
            "cisco-asa",
            "forwarded"
          ],
          "input" : {
            "type" : "log"
          },
          "@timestamp" : "2020-11-03T23:59:04.000+01:00",
          "ecs" : {
            "version" : "1.5.0"
          },
          "service" : {
            "type" : "cisco"
          },
          "host" : {
            "hostname" : "name",
            "os" : {
              "kernel" : "value",
              "codename" : "Core",
              "name" : "CentOS Linux",
              "family" : "redhat",
              "version" : "7 (Core)",
              "platform" : "centos"
            },
            "containerized" : false,
            "ip" : [
              "ip-1"
            ],
            "id" : "41c28f6d08964fcab092cb65e2cc5c18",
            "mac" : [
              "00:00:00:00:00:00"
            ],
            "architecture" : "x86_64"
          },
          "event" : {
            "severity" : 6,
            "timezone" : "+01:00",
            "module" : "cisco",
            "action" : "flow-expiration",
            "dataset" : "cisco.asa"
          },
          "cisco" : {
            "asa" : {
              "message_id" : "302014"
            }
          }
        },
        "sort" : [
          1604444344000
        ]
      }
    ]
  }
}

image
image992×646 67.2 KB

image
image1175×348 21 KB

I have just noticed that this probably is not taking form field I thought It was.

4

Oh wait, so that's an issue. Please, can you also open a GH issue here https://github.com/elastic/beats/issues so that someone fixes it?

By the way, about the yaml indentation, Filebeat won't give an error if your yaml is incorrect (correct YAML syntax but incorrect Filebeat I mean). It will simply think that the setting isn't there.

5

One last thing because after looking at this again, there's something I don't understand: Where do you get that message?

Your Cisco message is, for example,
Teardown TCP connection 23913857 for name-1:ip-2/49903 to name-2:ip-3/23 duration 25:54:08 bytes 4095435 TCP FINs.

And your error.message like:
Provided Grok expressions do not match field value: [Teardown TCP connection 23913857 for name-1:ip-2/49903 to name-2:ip-3/23 duration 25:54:08 bytes 4095435 TCP FINs]

So, for example, you can drop messages that contains Define processors | Filebeat Reference [8.11] | Elastic an error.message field.

6

image

image
image798×378 31.6 KB

I have change It to

and added field in logs

image
image1824×671 92.5 KB

processors:
  - drop_event:
         when:
         and:
         - equals.event.module: "cisco"
         - has_fields: ['error.message']

I don't get this output at all right now.

7

Sorry I didn't explain myself properly. I mean where the message is produced. Is this message produced in Filebeat Cisco module?

8

I am sending directly from filebeat to elastic.

As Kiban stands this "Message field is derived from document fields"

image
image1253×478 26.5 KB

9

It's working the last document with this field I got from

"@timestamp" : "2020-11-04T16:52:58.000+01:00",
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.