# Logstash mutate can not get variable

**URL:** <https://discuss.elastic.co/t/logstash-mutate-can-not-get-variable/359516>\
**Category:** Logstash\
**Created:** [May 15, 2024, 8:20am UTC](https://discuss.elastic.co/t/logstash-mutate-can-not-get-variable/359516 "2024-05-15T08:20:24Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Fodesu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fodesu/32/134439_2.png) [@Fodesu](https://discuss.elastic.co/u/Fodesu)\
**Post date:** [May 15, 2024, 8:20am UTC](https://discuss.elastic.co/t/logstash-mutate-can-not-get-variable/359516/1 "2024-05-15T08:20:24Z")

</div>

Hi Term,  
i am trying to make a grok parser for every two lines of log, i config filebeat.yml to parsing every two lines in filebeat. but it can't parser variable successful.

I use [grokdebug](https://grokdebugger.com?pattern=%5C%5B%5Cd%7B4%7D%5C%2F%5Cd%7B2%7D%5C%2F%5Cd%7B2%7D%5Cs*%5Cd%7B2%7D%3A%5Cd%7B2%7D%3A%5Cd%7B2%7D.%5Cd%7B6%7D%2C%5Cs*%25%7BNUMBER%3Aloglevel%7D%2C%5Cs*pid%3D%25%7BNUMBER%3Apid%7D%2C%20effective%5C(%25%7BNUMBER%3Aeff1%7D%2C%5Cs*%25%7BNUMBER%3Aeff2%7D%5C)%2C%5Cs*real%5C(%25%7BNUMBER%3Areal1%7D%2C%5Cs*%25%7BNUMBER%3Areal2%7D%5C)%2C%5Cs*class%3D%25%7BGREEDYDATA%3Aclass%7D%5C%5D%5Cs*%25%7BGREEDYDATA%3Asourcelocation%7D%3A%5Cs*%25%7BNUMBER%3Asource_line%7D%5C(%25%7BGREEDYDATA%3Afunction%7D%5C)%5Cs*%5C%5Cn%5Cs*%25%7BGREEDYDATA%3Amessage%7D&sample=%5B2024%2F05%2F12%2015%3A27%3A55.490453%2C%2010%2C%20pid%3D49544%2C%20effective(0%2C%200)%2C%20real(0%2C%200)%2C%20class%3Dsmb2%5D%20..%2F..%2Fsource3%2Fsmbd%2Fsmb2_server.c%3A2979(smbd_smb2_request_dispatch)%5Cn%20%20smbd_smb2_request_dispatch%3A%20opcode%5BSMB2_OP_NEGPROT%5D%20mid%20%3D%200)

Configuration is follow , Can anyone please help me solve this problem?  
filebeat config

```auto
filebeat.inputs:
- type: filestream
  id: my-filestream-id
  enabled: true
  paths:
    - /tmp/smb2.log
  parsers:
  - multiline:
      type: pattern
      pattern: '^\['
      negate: true
      match: after

```

logstash config

```auto
input {
        beats {
                port => 5044
        }
        stdin{}
}
filter {
        grok {
                patterns_dir => ["./patterns"]
                match => {
                        message => "\[\d{4}\/\d{2}\/\d{2}\s*\d{2}:\d{2}:\d{2}.\d{6},\s*%{NUMBER:loglevel},\s*pid=%{NUMBER:pid}, effective\(%{NUMBER:eff1},\s*%{NUMBER:eff2}\),\s*real\(%{NUMBER:real1},\s*%{NUMBER:real2}\),\s*class=%{GREEDYDATA:class}\]\s*%{GREEDYDATA:sourcelocation}:\s*%{NUMBER:source_line}\(%{GREEDYDATA:function}\)\s*\\n\s*%{GREEDYDATA:log_content}"
                }
        }

        mutate {
                add_field => {
                        "loglevel" => "%{loglevel}"
                        "pid" => "%{pid}"
                        "effective_num1" => "%{eff1}"
                        "effective_num2" => "%{eff2}"
                        "real_num1" => "%{real1}"
                        "real_num2" => "%{real2}"
                        "class" => "%{class}"
                        "sourcelocation" => "%{sourcelocation}"
                        "source_line" => "%{source_line}"
                        "function" => "%{function}"
                        "log_content" => "%{log_content}"
                }
        }
}
output {
        stdout{}
        file {
                path => "/tmp/output.json"
                codec => json
        }
}

```

log , two line once parse

```auto
[2024/05/12 15:27:55.489184, 10, pid=49544, effective(0, 0), real(0, 0), class=smb2] ../../source3/smbd/smb2_server.c:4531(smbd_smb2_process_negprot)
  smbd_smb2_first_negprot: packet length 236
[2024/05/12 15:27:55.490453, 10, pid=49544, effective(0, 0), real(0, 0), class=smb2] ../../source3/smbd/smb2_server.c:2979(smbd_smb2_request_dispatch)
  smbd_smb2_request_dispatch: opcode[SMB2_OP_NEGPROT] mid = 0

```

output, can not use variable

```auto
{
             "class" => "%{class}",
       "source_line" => "%{source_line}",
       "log_content" => "%{log_content}",
         "real_num1" => "%{real1}",
          "function" => "%{function}",
               "pid" => "%{pid}",
              "host" => {
                   "id" => "56973fcee2a14647a320e18be6b43f6b",
                   "ip" => [
            [0] "10.0.2.15",
            [1] "fe80::a00:27ff:fe56:a622",
            [2] "192.168.56.106",
            [3] "fe80::a00:27ff:fe8d:2c83"
        ],
                   "os" => {
              "kernel" => "5.15.0-106-generic",
             "version" => "22.04.4 LTS (Jammy Jellyfish)",
              "family" => "debian",
            "platform" => "ubuntu",
            "codename" => "jammy",
                "name" => "Ubuntu",
                "type" => "linux"
        },
                  "mac" => [
            [0] "08:00:27:56:a6:22",
            [1] "08:00:27:8d:2c:83"
        ],
                 "name" => "momo",
         "architecture" => "x86_64",
        "containerized" => false,
             "hostname" => "momo"
    },
    "sourcelocation" => "%{sourcelocation}",
    "effective_num2" => "%{eff2}",
           "message" => "[2024/05/12 15:27:55.489184, 10, pid=49544, effective(0, 0), real(0, 0), class=smb2] ../../source3/smbd/smb2_server.c:4531(smbd_smb2_process_negprot)\n smbd_smb2_first_negprot: packet length 236",
        "@timestamp" => 2024-05-15T07:53:09.750Z,
              "tags" => [
        [0] "beats_input_codec_plain_applied",
        [1] "_grokparsefailure"
    ],
             "agent" => {
             "version" => "7.17.21",
                  "id" => "90964887-b3a5-4740-b0c1-48ccc07fb8d9",
        "ephemeral_id" => "ca791cf8-a014-440f-90b1-0d8f070fc345",
                "name" => "momo",
                "type" => "filebeat",
            "hostname" => "momo"
    },
               "ecs" => {
        "version" => "1.12.0"
    },
    "effective_num1" => "%{eff1}",
             "input" => {
        "type" => "filestream"
    },
         "real_num2" => "%{real2}",
          "loglevel" => "%{loglevel}",
          "@version" => "1",
               "log" => {
        "offset" => 150,
         "flags" => [
            [0] "multiline"
        ],
          "file" => {
            "path" => "/tmp/smb2.log"
        }
    }
}

```

---

<div class="post-metadata">

**Author:** ![Fodesu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fodesu/32/134439_2.png) [@Fodesu](https://discuss.elastic.co/u/Fodesu)\
**Post date:** [May 15, 2024, 9:37am UTC](https://discuss.elastic.co/t/logstash-mutate-can-not-get-variable/359516/2 "2024-05-15T09:37:07Z")

</div>

can someone help me? 🙃

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 15, 2024, 1:51pm UTC](https://discuss.elastic.co/t/logstash-mutate-can-not-get-variable/359516/3 "2024-05-15T13:51:40Z")

</div>

> [@Fodesu](#):
>
> ```auto
> "tags" => [
> [0] "beats_input_codec_plain_applied",
> [1] "_grokparsefailure"
> ],
> 
> ```

Your grok pattern is broken

```
\(%{GREEDYDATA:function}\)\s*\\n\s*%{GREEDYDATA:log_content}

```

should be

```
\(%{GREEDYDATA:function}\)\s*\n\s*%{GREEDYDATA:log_content}

```

Several of those GREEDYDATA patterns could be replaced with NOTSPACE, which would be faster, especially when an event does not match the pattern.

It is unclear why you are doing this:

```
mutate {
    add_field => {
        "source_line" => "%{source_line}"
        etc.

```

That takes

```
   "source_line" => "2979",

```

and converts it to

```
   "source_line" => [
    [0] "2979",
    [1] "2979"
],

```

which does not look useful.

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [May 15, 2024, 1:58pm UTC](https://discuss.elastic.co/t/logstash-mutate-can-not-get-variable/359516/4 "2024-05-15T13:58:54Z")

</div>

Badger has already suggested the solution. This is slightly different with DATA or just use NOTSPACE what Badger suggested.

`match => {"message" => "\[\d{4}\/\d{2}\/\d{2}\s*\d{2}:\d{2}:\d{2}.\d{6},\s*%{NUMBER:loglevel},\s*pid=%{NUMBER:pid}, effective\(%{NUMBER:eff1},\s*%{NUMBER:eff2}\),\s*real\(%{NUMBER:real1},\s*%{NUMBER:real2}\),\s*class=%{DATA:class}\]\s*%{DATA:sourcelocation}:\s*%{NUMBER:source_line}\(%{DATA:function}\)\n\s*%{GREEDYDATA:log_content}" }`

And yes Badger, I had been also confused why my grok is adding double data, until noticed mutate add\_field 🙂