Filebeat processor not doing anything

Elastic Stack Elasticsearch
1

I'm trying to use a processor to split up syslog messages into separate fields (using the '=' character as a delimiter). Here's my processor:

- type: syslog
  format: auto
  protocol.udp:
    host: "0.0.0.0:9002"
  tags: ["filebeat-syslog"]
  processors:
    - if:
        equals:
          input.type: "syslog"
      then:
        - dissect:
            tokenizer: '"%{log.preamble}=%{log.sourceip}"'
            field: "message"
            target_prefix: "dissect"

Here's an example of a log file that I'm feeding it:

{
  "_index": ".ds-filebeat-8.9.1-2023.09.11-000009",
  "_id": "CjBmi4oBzDe7dmnI5VCr",
  "_version": 1,
  "_score": 0,
  "_source": {
    "@timestamp": "2023-09-12T21:58:02.332Z",
    "message": "<140>Original Address=[redacted] Sep 12 17:58:02 2023 [redacted]cluster_mgr[4306]: <352310> <4306> <WARN> <[redacted]>  (cm_update_l2_connectivity) Peer [redacted] Connection status L3 Connected, Failed VLAN 814",
    "log": {
      "source": {
        "address": "[redacted]:64187"
      }
    },
    "tags": [
      "syslog"
    ],
    "input": {
      "type": "syslog"
    },
    "agent": {
      "ephemeral_id": "9047818c-71f4-4b24-9bb0-f58fa609f5af",
      "id": "998ebe76-3f5e-48e2-9944-e1ba6df5656f",
      "name": "zeek1",
      "type": "filebeat",
      "version": "8.9.1"
    },
    "ecs": {
      "version": "8.0.0"
    },
    "host": {
      "mac": [
        "18-66-DA-AC-80-07",
        "18-66-DA-AC-80-08",
        "18-66-DA-AC-80-09",
        "18-66-DA-AC-80-0A",
        "A0-36-9F-F0-EB-24",
        "A0-36-9F-F0-EB-26",
        "A0-36-9F-F0-F1-40",
        "A0-36-9F-F0-F1-42"
      ],
      "hostname": "zeek1",
      "architecture": "x86_64",
      "os": {
        "codename": "jammy",
        "type": "linux",
        "platform": "ubuntu",
        "version": "22.04.3 LTS (Jammy Jellyfish)",
        "family": "debian",
        "name": "Ubuntu",
        "kernel": "5.15.0-78-generic"
      },
      "id": "5eb37bd26b0e4ba792a1575c41c3a2db",
      "containerized": false,
      "name": "zeek1",
      "ip": [
        "[redacted]",
        "fe80::1a66:daff:feac:8008",
        "fe80::1a66:daff:feac:8009",
        "fe80::a236:9fff:fef0:f140",
        "fe80::a236:9fff:fef0:f142",
        "fe80::a236:9fff:fef0:eb26"
      ]
    }
  },
  "fields": {
    "host.os.name.text": [
      "Ubuntu"
    ],
    "host.hostname": [
      "zeek1"
    ],
    "host.mac": [
      "18-66-DA-AC-80-07",
      "18-66-DA-AC-80-08",
      "18-66-DA-AC-80-09",
      "18-66-DA-AC-80-0A",
      "A0-36-9F-F0-EB-24",
      "A0-36-9F-F0-EB-26",
      "A0-36-9F-F0-F1-40",
      "A0-36-9F-F0-F1-42"
    ],
    "host.ip": [
      "[redacted]",
      "fe80::1a66:daff:feac:8008",
      "fe80::1a66:daff:feac:8009",
      "fe80::a236:9fff:fef0:f140",
      "fe80::a236:9fff:fef0:f142",
      "fe80::a236:9fff:fef0:eb26"
    ],
    "agent.type": [
      "filebeat"
    ],
    "host.os.version": [
      "22.04.3 LTS (Jammy Jellyfish)"
    ],
    "host.os.kernel": [
      "5.15.0-78-generic"
    ],
    "host.os.name": [
      "Ubuntu"
    ],
    "agent.name": [
      "zeek1"
    ],
    "host.name": [
      "zeek1"
    ],
    "host.id": [
      "5eb37bd26b0e4ba792a1575c41c3a2db"
    ],
    "host.os.type": [
      "linux"
    ],
    "host.os.codename": [
      "jammy"
    ],
    "input.type": [
      "syslog"
    ],
    "agent.hostname": [
      "zeek1"
    ],
    "message": [
      "<140>Original Address=[redacted] Sep 12 17:58:02 2023 [redacted] cluster_mgr[4306]: <352310> <4306> <WARN> <[redacted]>  (cm_update_l2_connectivity) Peer [redacted] Connection status L3 Connected, Failed VLAN 814"
    ],
    "tags": [
      "filebeat-syslog"
    ],
    "host.architecture": [
      "x86_64"
    ],
    "@timestamp": [
      "2023-09-12T21:58:02.332Z"
    ],
    "agent.id": [
      "998ebe76-3f5e-48e2-9944-e1ba6df5656f"
    ],
    "ecs.version": [
      "8.0.0"
    ],
    "host.containerized": [
      false
    ],
    "host.os.platform": [
      "ubuntu"
    ],
    "log.source.address": [
      "[redacted]:64187"
    ],
    "agent.ephemeral_id": [
      "9047818c-71f4-4b24-9bb0-f58fa609f5af"
    ],
    "agent.version": [
      "8.9.1"
    ],
    "host.os.family": [
      "debian"
    ]
  }
}

However, the log remains unchanged...

2

Hi @artschooldropout

Curious why you are putting the if in there instead of just the dissect since that will only be applied to this input (this is an input specific process not a global processor

- type: syslog
  format: auto
  protocol.udp:
    host: "0.0.0.0:9002"
  tags: ["filebeat-syslog"]
  processors:
    - dissect:
        tokenizer: '"%{log.preamble}=%{log.sourceip}"'
        field: "message"
        target_prefix: "dissect"
3

Aha - ok, that's my misunderstanding. I tried as you suggested, and I get a somewhat ambiguous flag:

log.flags   dissect_parsing_error
4

By the way, I tried this fun little 'Dissect tester' site: Test Tokenizer Patterns for the Dissect filter

The site shows a nicely tokenized output:

{
  "syslog.preamble": "<140>Original Address",
  "syslog.sourceip": "[redacted] Sep 12 17:49:48 2023 [redacted] cluster_mgr[4306]: <352310> <4306>    (cm_update_l2_connectivity) Peer [redacted] Connection status L3 Connected, Failed VLAN 814"
}
5

Aha - I had extra single quotation marks around my dissector. It should have been:

 tokenizer: "%{log.preamble}=%{log.sourceip}"

Thanks again @stephenb for your help!

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.