Logstash mutate can not get variable

Elastic Stack Logstash
1

Hi Term,
i am trying to make a grok parser for every two lines of log, i config filebeat.yml to parsing every two lines in filebeat. but it can't parser variable successful.

I use grokdebug

Configuration is follow , Can anyone please help me solve this problem?
filebeat config

filebeat.inputs:
- type: filestream
  id: my-filestream-id
  enabled: true
  paths:
    - /tmp/smb2.log
  parsers:
  - multiline:
      type: pattern
      pattern: '^\['
      negate: true
      match: after

logstash config

input {
        beats {
                port => 5044
        }
        stdin{}
}
filter {
        grok {
                patterns_dir => ["./patterns"]
                match => {
                        message => "\[\d{4}\/\d{2}\/\d{2}\s*\d{2}:\d{2}:\d{2}.\d{6},\s*%{NUMBER:loglevel},\s*pid=%{NUMBER:pid}, effective\(%{NUMBER:eff1},\s*%{NUMBER:eff2}\),\s*real\(%{NUMBER:real1},\s*%{NUMBER:real2}\),\s*class=%{GREEDYDATA:class}\]\s*%{GREEDYDATA:sourcelocation}:\s*%{NUMBER:source_line}\(%{GREEDYDATA:function}\)\s*\\n\s*%{GREEDYDATA:log_content}"
                }
        }

        mutate {
                add_field => {
                        "loglevel" => "%{loglevel}"
                        "pid" => "%{pid}"
                        "effective_num1" => "%{eff1}"
                        "effective_num2" => "%{eff2}"
                        "real_num1" => "%{real1}"
                        "real_num2" => "%{real2}"
                        "class" => "%{class}"
                        "sourcelocation" => "%{sourcelocation}"
                        "source_line" => "%{source_line}"
                        "function" => "%{function}"
                        "log_content" => "%{log_content}"
                }
        }
}
output {
        stdout{}
        file {
                path => "/tmp/output.json"
                codec => json
        }
}

log , two line once parse

[2024/05/12 15:27:55.489184, 10, pid=49544, effective(0, 0), real(0, 0), class=smb2] ../../source3/smbd/smb2_server.c:4531(smbd_smb2_process_negprot)
  smbd_smb2_first_negprot: packet length 236
[2024/05/12 15:27:55.490453, 10, pid=49544, effective(0, 0), real(0, 0), class=smb2] ../../source3/smbd/smb2_server.c:2979(smbd_smb2_request_dispatch)
  smbd_smb2_request_dispatch: opcode[SMB2_OP_NEGPROT] mid = 0

output, can not use variable

{
             "class" => "%{class}",
       "source_line" => "%{source_line}",
       "log_content" => "%{log_content}",
         "real_num1" => "%{real1}",
          "function" => "%{function}",
               "pid" => "%{pid}",
              "host" => {
                   "id" => "56973fcee2a14647a320e18be6b43f6b",
                   "ip" => [
            [0] "10.0.2.15",
            [1] "fe80::a00:27ff:fe56:a622",
            [2] "192.168.56.106",
            [3] "fe80::a00:27ff:fe8d:2c83"
        ],
                   "os" => {
              "kernel" => "5.15.0-106-generic",
             "version" => "22.04.4 LTS (Jammy Jellyfish)",
              "family" => "debian",
            "platform" => "ubuntu",
            "codename" => "jammy",
                "name" => "Ubuntu",
                "type" => "linux"
        },
                  "mac" => [
            [0] "08:00:27:56:a6:22",
            [1] "08:00:27:8d:2c:83"
        ],
                 "name" => "momo",
         "architecture" => "x86_64",
        "containerized" => false,
             "hostname" => "momo"
    },
    "sourcelocation" => "%{sourcelocation}",
    "effective_num2" => "%{eff2}",
           "message" => "[2024/05/12 15:27:55.489184, 10, pid=49544, effective(0, 0), real(0, 0), class=smb2] ../../source3/smbd/smb2_server.c:4531(smbd_smb2_process_negprot)\n  smbd_smb2_first_negprot: packet length 236",
        "@timestamp" => 2024-05-15T07:53:09.750Z,
              "tags" => [
        [0] "beats_input_codec_plain_applied",
        [1] "_grokparsefailure"
    ],
             "agent" => {
             "version" => "7.17.21",
                  "id" => "90964887-b3a5-4740-b0c1-48ccc07fb8d9",
        "ephemeral_id" => "ca791cf8-a014-440f-90b1-0d8f070fc345",
                "name" => "momo",
                "type" => "filebeat",
            "hostname" => "momo"
    },
               "ecs" => {
        "version" => "1.12.0"
    },
    "effective_num1" => "%{eff1}",
             "input" => {
        "type" => "filestream"
    },
         "real_num2" => "%{real2}",
          "loglevel" => "%{loglevel}",
          "@version" => "1",
               "log" => {
        "offset" => 150,
         "flags" => [
            [0] "multiline"
        ],
          "file" => {
            "path" => "/tmp/smb2.log"
        }
    }
}
2

can someone help me? :upside_down_face:

3

Your grok pattern is broken

\(%{GREEDYDATA:function}\)\s*\\n\s*%{GREEDYDATA:log_content}

should be

\(%{GREEDYDATA:function}\)\s*\n\s*%{GREEDYDATA:log_content}

Several of those GREEDYDATA patterns could be replaced with NOTSPACE, which would be faster, especially when an event does not match the pattern.

It is unclear why you are doing this:

mutate {
    add_field => {
        "source_line" => "%{source_line}"
        etc.

That takes

   "source_line" => "2979",

and converts it to

   "source_line" => [
    [0] "2979",
    [1] "2979"
],

which does not look useful.

1 Like
4

Badger has already suggested the solution. This is slightly different with DATA or just use NOTSPACE what Badger suggested.

match => {"message" => "\[\d{4}\/\d{2}\/\d{2}\s*\d{2}:\d{2}:\d{2}.\d{6},\s*%{NUMBER:loglevel},\s*pid=%{NUMBER:pid}, effective\(%{NUMBER:eff1},\s*%{NUMBER:eff2}\),\s*real\(%{NUMBER:real1},\s*%{NUMBER:real2}\),\s*class=%{DATA:class}\]\s*%{DATA:sourcelocation}:\s*%{NUMBER:source_line}\(%{DATA:function}\)\n\s*%{GREEDYDATA:log_content}" }

And yes Badger, I had been also confused why my grok is adding double data, until noticed mutate add_field :slight_smile: