Logstash filter mutate problem

Elastic Stack Logstash
1

I have created a filter as shown below

filter {
    if [application] == "today" {
       if field1 {
           mutate {
            add_field => {
               mynewfield => "%{[field1]}"
                }
              }
            }
       if [event][module] {
           mutate {
               add_field => {
                   mew => "%{[mynewfield]}"
                }
             }
         }
      }
  }
}

the variable "mew" is not showing the value of 'mynewfield' instead it's showing %{[mynewfield]} in kibana. how do I resolve this please.

Regards

2

If it is showing the literal value of %{[mynewfield]} this would mean that the condtional if [event][module] is not matching, which means that your event does not have the field event.module.

Please share an event where you think this should match but it is not matching to show what fields are present in the document.

3

image
image1000×580 111 KB

4

The message you shared does not have the issue you mentioned, it does not have the mew field.

Please share a message where you have both the application and the mew field.

Also, share the json contents of the message, just expand the document in kibana and copy the content of the json tab, use the preformatted text option to share.

5 
{
  "_index": "auditbeat-2023.07.31",
  "_type": "_doc",
  "_id": "KhknrIkBBEGDHOFEynSE",
  "_version": 1,
  "_score": null,
  "_source": {
    "ecs": {
      "version": "1.1.0"
    },
    "@version": "1",
    "my_client_ip": "%{[system][socket][client][ip]}",
    "my_newfield1": "%{my_newfield}",
    "tag_1": "beats123",
    "agent": {
      "ephemeral_id": "de04927d-1df2-4c28-9831-e234b8239c26",
      "hostname": "userB",
      "type": "auditbeat",
      "id": "1b2ffabb-287b-4a21-8e9e-7886e49673b0",
      "version": "7.4.1"
    },
    "signature": "auditbeat",
    "message": "Process sendmail-mta (PID: 20289) by user root STARTED",
    "@timestamp": "2023-07-31T13:33:49.143Z",
    "cloud": {
      "machine": {
        "type": "standard.medium"
      },
      "instance": {
        "id": "i-0000021a",
        "name": "default0-p0000000044-s0000000048-userb.novalocal"
      },
      "provider": "openstack",
      "availability_zone": "nova"
    },
    "application": "auditbeat",
    "user": {
      "saved": {
        "id": "0",
        "group": {
          "id": "119"
        }
      },
      "name": "root",
      "group": {
        "id": "0",
        "name": "root"
      },
      "id": "0",
      "effective": {
        "id": "0",
        "group": {
          "id": "119"
        }
      }
    },
    "tag_2": "beats_input_codec_plain_applied",
    "host": {
      "name": "userB",
      "hostname": "userB",
      "containerized": false,
      "architecture": "x86_64",
      "os": {
        "name": "Ubuntu",
        "platform": "ubuntu",
        "version": "14.04.6 LTS, Trusty Tahr",
        "codename": "trusty",
        "kernel": "3.13.0-24-generic",
        "family": "debian"
      },
      "id": "26b1ceca54c197067320f0bf634024c6"
    },
    "process": {
      "name": "sendmail-mta",
      "ppid": 2387,
      "start": "2023-07-31T13:33:42.190Z",
      "entity_id": "TP065Y1SDJ30ZqCD",
      "args": [
        "sendmail: MTA: 36VDXgjq020289 localhost [127.0.0.1]: MAIL FROM"
      ],
      "hash": {
        "sha1": "59a52509020b6fe10ac9585453c9138732879b7f"
      },
      "working_directory": "/var/spool/mqueue",
      "pid": 20289,
      "executable": "/usr/lib/sm.bin/sendmail"
    },
    "service": {
      "type": "system"
    },
    "event": {
      "kind": "event",
      "module": "system",
      "dataset": "process",
      "action": "process_started"
    }
  },
  "fields": {
    "@timestamp": [
      "2023-07-31T13:33:49.143Z"
    ],
    "process.start": [
      "2023-07-31T13:33:42.190Z"
    ]
  },
  "highlight": {
    "event.module": [
      "@kibana-highlighted-field@system@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1690810429143
  ]
}
6

The "mew" is the same thing as "newfield1" in the above json.
"mynewfield" gets the actual value of "field1"

7

You need to share the exact document where you are having this issue, do not change fields names as this will lead to confusion.

The document you share does not match your issue, it does not have any field named mew and the field application is not equal to today, so this is unrelated to the issue in your first post.

Can you share a document where you have the application equals to today and the field mew exists and it is equal to mynewfield? Or did you change those values?

This issue is pretty common, when you have the literal value of the field when using the add_field in mutate, this means that the source field does not exist in the document, so you need to check your conditionals.

8 
filter {
    if [application] == "auditbeat" {
       if [auditd][summary][object][primary] {
           mutate {
            add_field => {
               my_newfield => "%{[auditd][summary][object][primary]}"
                }
              }
            }
        }
   }
filter {
   if [application] == "auditbeat" {
       if [event][module] == "system" {
           mutate {
               add_field => {
                   my_newfield1 => "%{[my_newfield]}"
                }
             }
         }
      }
  }
}
9

Well, the document you shared does not have the field [auditd][summary][object][primary], so the mutate is working as expected.

Since the field [auditd][summary][object][primary] does not exist, the field my_newfield will not be created.

Your second conditional does not check if the field my_newfield exists, it just checks if the application is equal to auditbeat and if the [event][module] field is equal to system, in the sample document you shared both conditions are true, so the mutate field will execute the add_field, but the field my_newfield does not exist, then you will get its literal value.

I see no issue, the mutate is working as expected according to your conditionals, you need to validated them and check if the my_newfield exists before using it as a source in another mutate.

10

the "my_newfield" is created from this json document

{
  "_index": "auditbeat-2023.07.31",
  "_type": "_doc",
  "_id": "aRlFrIkBBEGDHOFEcKA2",
  "_version": 1,
  "_score": null,
  "_source": {
    "event": {
      "action": "connected-to",
      "module": "auditd",
      "outcome": "success",
      "category": "audit-rule"
    },
    "ecs": {
      "version": "1.1.0"
    },
    "auditd": {
      "data": {
        "a0": "5",
        "tty": "(none)",
        "arch": "x86_64",
        "a1": "1c61150",
        "syscall": "connect",
        "a3": "d866133124f88",
        "socket": {
          "addr": "10.0.2.24",
          "port": "53",
          "family": "ipv4"
        },
        "exit": "0",
        "a2": "10"
      },
      "sequence": 4192387,
      "result": "success",
      "message_type": "syscall",
      "summary": {
        "how": "/usr/lib/sm.bin/sendmail",
        "actor": {
          "secondary": "root",
          "primary": "unset"
        },
        "object": {
          "secondary": "53",
          "primary": "10.0.2.24",
          "type": "socket"
        }
      }
    },
    "my_client_ip": "%{[system][socket][client][ip]}",
    "network": {
      "direction": "outgoing"
    },
    "agent": {
      "ephemeral_id": "de04927d-1df2-4c28-9831-e234b8239c26",
      "type": "auditbeat",
      "hostname": "userB",
      "id": "1b2ffabb-287b-4a21-8e9e-7886e49673b0",
      "version": "7.4.1"
    },
    "@version": "1",
    "signature": "auditbeat",
    "tag_1": "external-access",
    "tag_3": "beats_input_raw_event",
    "@timestamp": "2023-07-31T14:06:12.128Z",
    "user": {
      "name": "root",
      "group": {
        "id": "0",
        "name": "root"
      },
      "saved": {
        "name": "root",
        "group": {
          "id": "119",
          "name": "smmsp"
        },
        "id": "0"
      },
      "filesystem": {
        "name": "root",
        "group": {
          "id": "119",
          "name": "smmsp"
        },
        "id": "0"
      },
      "id": "0",
      "effective": {
        "name": "root",
        "group": {
          "id": "119",
          "name": "smmsp"
        },
        "id": "0"
      }
    },
    "cloud": {
      "machine": {
        "type": "standard.medium"
      },
      "instance": {
        "id": "i-0000021a",
        "name": "default0-p0000000044-s0000000048-userb.novalocal"
      },
      "provider": "openstack",
      "availability_zone": "nova"
    },
    "application": "auditbeat",
    "tag_2": "beats123",
    "host": {
      "hostname": "userB",
      "name": "userB",
      "containerized": false,
      "architecture": "x86_64",
      "os": {
        "name": "Ubuntu",
        "platform": "ubuntu",
        "version": "14.04.6 LTS, Trusty Tahr",
        "codename": "trusty",
        "kernel": "3.13.0-24-generic",
        "family": "debian"
      },
      "id": "26b1ceca54c197067320f0bf634024c6"
    },
    "service": {
      "type": "auditd"
    },
    "process": {
      "executable": "/usr/lib/sm.bin/sendmail",
      "name": "sendmail-mta",
      "pid": 21089,
      "ppid": 2387
    },
    "my_newfield": "10.0.2.24",
    "destination": {
      "port": "53",
      "ip": "10.0.2.24"
    }
  },
  "fields": {
    "@timestamp": [
      "2023-07-31T14:06:12.128Z"
    ]
  },
  "highlight": {
    "event.module": [
      "@kibana-highlighted-field@auditd@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1690812372128
  ]
}
11

As you can see there are two json document and I need to pass the field of one document into another document. Or possibly can I combine both documents to produce a new document? and if yes how do I do that.

Thanks for your swift response

Regards

12

Then this is also expected, Logstash events are independent from each other, so if you created a field on one document, it will only exists on that document, you can't use this field on other events.

What you can try to do is to use the aggregate filter to combine two events and create one document, you need to have a unique identificar in both events to be able to combine them.

As mentioned before, you can try to use the aggregate filter to combine two events, you would need to have a unique identificar in both events to combine them and you will also need to run the pipeline with just one worker.

I do not use the aggregate filter so I can not help further, but there are a couple of messages in the forum about it that may help your use case.

I suggest you open a new topic as this is a completely different issue.

13

thanks

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.