# Logstash filter mutate problem
**URL:** https://discuss.elastic.co/t/logstash-filter-mutate-problem/339690
**Category:** Logstash
**Created:** [July 31, 2023, 12:15pm UTC](https://discuss.elastic.co/t/logstash-filter-mutate-problem/339690 "2023-07-31T12:15:21Z")
**Posts on this page:** 14
**Page:** 1
### Author:  [@pero](https://discuss.elastic.co/u/pero)
#### Post date: [July 31, 2023, 12:15pm UTC](https://discuss.elastic.co/t/logstash-filter-mutate-problem/339690/1 "2023-07-31T12:15:21Z")
I have created a filter as shown below
```auto
filter {
if [application] == "today" {
if field1 {
mutate {
add_field => {
mynewfield => "%{[field1]}"
}
}
}
if [event][module] {
mutate {
add_field => {
mew => "%{[mynewfield]}"
}
}
}
}
}
}
```
the variable "mew" is not showing the value of 'mynewfield' instead it's showing %{[mynewfield]} in kibana. how do I resolve this please.
Regards
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [July 31, 2023, 12:44pm UTC](https://discuss.elastic.co/t/logstash-filter-mutate-problem/339690/2 "2023-07-31T12:44:31Z")
> [@pero](#):
>
> the variable "mew" is not showing the value of 'mynewfield' instead it's showing %{[mynewfield]} in kibana. how do I resolve this please.
If it is showing the literal value of `%{[mynewfield]}` this would mean that the condtional `if [event][module]` is not matching, which means that your event does not have the field `event.module`.
Please share an event where you think this should match but it is not matching to show what fields are present in the document.
---
### Author:  [@pero](https://discuss.elastic.co/u/pero)
#### Post date: [July 31, 2023, 12:51pm UTC](https://discuss.elastic.co/t/logstash-filter-mutate-problem/339690/3 "2023-07-31T12:51:04Z")
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [July 31, 2023, 1:04pm UTC](https://discuss.elastic.co/t/logstash-filter-mutate-problem/339690/4 "2023-07-31T13:04:00Z")
The message you shared does not have the issue you mentioned, it does not have the `mew` field.
Please share a message where you have both the `application` and the `mew` field.
Also, share the _json_ contents of the message, just expand the document in kibana and copy the content of the _json_ tab, use the preformatted text option to share.
---
### Author:  [@pero](https://discuss.elastic.co/u/pero)
#### Post date: [July 31, 2023, 1:38pm UTC](https://discuss.elastic.co/t/logstash-filter-mutate-problem/339690/5 "2023-07-31T13:38:18Z")
```auto
{
"_index": "auditbeat-2023.07.31",
"_type": "_doc",
"_id": "KhknrIkBBEGDHOFEynSE",
"_version": 1,
"_score": null,
"_source": {
"ecs": {
"version": "1.1.0"
},
"@version": "1",
"my_client_ip": "%{[system][socket][client][ip]}",
"my_newfield1": "%{my_newfield}",
"tag_1": "beats123",
"agent": {
"ephemeral_id": "de04927d-1df2-4c28-9831-e234b8239c26",
"hostname": "userB",
"type": "auditbeat",
"id": "1b2ffabb-287b-4a21-8e9e-7886e49673b0",
"version": "7.4.1"
},
"signature": "auditbeat",
"message": "Process sendmail-mta (PID: 20289) by user root STARTED",
"@timestamp": "2023-07-31T13:33:49.143Z",
"cloud": {
"machine": {
"type": "standard.medium"
},
"instance": {
"id": "i-0000021a",
"name": "default0-p0000000044-s0000000048-userb.novalocal"
},
"provider": "openstack",
"availability_zone": "nova"
},
"application": "auditbeat",
"user": {
"saved": {
"id": "0",
"group": {
"id": "119"
}
},
"name": "root",
"group": {
"id": "0",
"name": "root"
},
"id": "0",
"effective": {
"id": "0",
"group": {
"id": "119"
}
}
},
"tag_2": "beats_input_codec_plain_applied",
"host": {
"name": "userB",
"hostname": "userB",
"containerized": false,
"architecture": "x86_64",
"os": {
"name": "Ubuntu",
"platform": "ubuntu",
"version": "14.04.6 LTS, Trusty Tahr",
"codename": "trusty",
"kernel": "3.13.0-24-generic",
"family": "debian"
},
"id": "26b1ceca54c197067320f0bf634024c6"
},
"process": {
"name": "sendmail-mta",
"ppid": 2387,
"start": "2023-07-31T13:33:42.190Z",
"entity_id": "TP065Y1SDJ30ZqCD",
"args": [
"sendmail: MTA: 36VDXgjq020289 localhost [127.0.0.1]: MAIL FROM"
],
"hash": {
"sha1": "59a52509020b6fe10ac9585453c9138732879b7f"
},
"working_directory": "/var/spool/mqueue",
"pid": 20289,
"executable": "/usr/lib/sm.bin/sendmail"
},
"service": {
"type": "system"
},
"event": {
"kind": "event",
"module": "system",
"dataset": "process",
"action": "process_started"
}
},
"fields": {
"@timestamp": [
"2023-07-31T13:33:49.143Z"
],
"process.start": [
"2023-07-31T13:33:42.190Z"
]
},
"highlight": {
"event.module": [
"@kibana-highlighted-field@system@/kibana-highlighted-field@"
]
},
"sort": [
1690810429143
]
}
```
---
### Author:  [@pero](https://discuss.elastic.co/u/pero)
#### Post date: [July 31, 2023, 1:43pm UTC](https://discuss.elastic.co/t/logstash-filter-mutate-problem/339690/6 "2023-07-31T13:43:22Z")
The "mew" is the same thing as "newfield1" in the above json.
"mynewfield" gets the actual value of "field1"
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [July 31, 2023, 1:45pm UTC](https://discuss.elastic.co/t/logstash-filter-mutate-problem/339690/7 "2023-07-31T13:45:46Z")
You need to share the exact document where you are having this issue, do not change fields names as this will lead to confusion.
The document you share does not match your issue, it does not have any field named `mew` and the field `application` is not equal to `today`, so this is unrelated to the issue in your first post.
Can you share a document where you have the `application` equals to `today` and the field `mew` exists and it is equal to `mynewfield`? Or did you change those values?
This issue is pretty common, when you have the literal value of the field when using the `add_field` in mutate, this means that the source field **does not** exist in the document, so you need to check your conditionals.
---
### Author:  [@pero](https://discuss.elastic.co/u/pero)
#### Post date: [July 31, 2023, 1:55pm UTC](https://discuss.elastic.co/t/logstash-filter-mutate-problem/339690/8 "2023-07-31T13:55:33Z")
> [@pero](#):
>
> ```auto
> filter {
> if [application] == "auditbeat" {
> if [auditd][summary][object][primary] {
> mutate {
> add_field => {
> my_newfield => "%{[auditd][summary][object][primary]}"
> }
> }
> }
> }
> }
> filter {
> if [application] == "auditbeat" {
> if [event][module] == "system" {
> mutate {
> add_field => {
> my_newfield1 => "%{[my_newfield]}"
> }
> }
> }
> }
> }
> }
>
> ```
```auto
filter {
if [application] == "auditbeat" {
if [auditd][summary][object][primary] {
mutate {
add_field => {
my_newfield => "%{[auditd][summary][object][primary]}"
}
}
}
}
}
filter {
if [application] == "auditbeat" {
if [event][module] == "system" {
mutate {
add_field => {
my_newfield1 => "%{[my_newfield]}"
}
}
}
}
}
}
```
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [July 31, 2023, 2:03pm UTC](https://discuss.elastic.co/t/logstash-filter-mutate-problem/339690/9 "2023-07-31T14:03:42Z")
Well, the document you shared does not have the field `[auditd][summary][object][primary]`, so the mutate is working as expected.
Since the field `[auditd][summary][object][primary]` does not exist, the field `my_newfield` will not be created.
Your second conditional does not check if the field `my_newfield` exists, it just checks if the `application` is equal to `auditbeat` and if the `[event][module]` field is equal to `system`, in the sample document you shared both conditions are true, so the mutate field will execute the `add_field`, but the field `my_newfield` does not exist, then you will get its literal value.
I see no issue, the mutate is working as expected according to your conditionals, you need to validated them and check if the `my_newfield` exists before using it as a source in another mutate.
---
### Author:  [@pero](https://discuss.elastic.co/u/pero)
#### Post date: [July 31, 2023, 2:08pm UTC](https://discuss.elastic.co/t/logstash-filter-mutate-problem/339690/10 "2023-07-31T14:08:46Z")
the "my\_newfield" is created from this json document
```auto
{
"_index": "auditbeat-2023.07.31",
"_type": "_doc",
"_id": "aRlFrIkBBEGDHOFEcKA2",
"_version": 1,
"_score": null,
"_source": {
"event": {
"action": "connected-to",
"module": "auditd",
"outcome": "success",
"category": "audit-rule"
},
"ecs": {
"version": "1.1.0"
},
"auditd": {
"data": {
"a0": "5",
"tty": "(none)",
"arch": "x86_64",
"a1": "1c61150",
"syscall": "connect",
"a3": "d866133124f88",
"socket": {
"addr": "10.0.2.24",
"port": "53",
"family": "ipv4"
},
"exit": "0",
"a2": "10"
},
"sequence": 4192387,
"result": "success",
"message_type": "syscall",
"summary": {
"how": "/usr/lib/sm.bin/sendmail",
"actor": {
"secondary": "root",
"primary": "unset"
},
"object": {
"secondary": "53",
"primary": "10.0.2.24",
"type": "socket"
}
}
},
"my_client_ip": "%{[system][socket][client][ip]}",
"network": {
"direction": "outgoing"
},
"agent": {
"ephemeral_id": "de04927d-1df2-4c28-9831-e234b8239c26",
"type": "auditbeat",
"hostname": "userB",
"id": "1b2ffabb-287b-4a21-8e9e-7886e49673b0",
"version": "7.4.1"
},
"@version": "1",
"signature": "auditbeat",
"tag_1": "external-access",
"tag_3": "beats_input_raw_event",
"@timestamp": "2023-07-31T14:06:12.128Z",
"user": {
"name": "root",
"group": {
"id": "0",
"name": "root"
},
"saved": {
"name": "root",
"group": {
"id": "119",
"name": "smmsp"
},
"id": "0"
},
"filesystem": {
"name": "root",
"group": {
"id": "119",
"name": "smmsp"
},
"id": "0"
},
"id": "0",
"effective": {
"name": "root",
"group": {
"id": "119",
"name": "smmsp"
},
"id": "0"
}
},
"cloud": {
"machine": {
"type": "standard.medium"
},
"instance": {
"id": "i-0000021a",
"name": "default0-p0000000044-s0000000048-userb.novalocal"
},
"provider": "openstack",
"availability_zone": "nova"
},
"application": "auditbeat",
"tag_2": "beats123",
"host": {
"hostname": "userB",
"name": "userB",
"containerized": false,
"architecture": "x86_64",
"os": {
"name": "Ubuntu",
"platform": "ubuntu",
"version": "14.04.6 LTS, Trusty Tahr",
"codename": "trusty",
"kernel": "3.13.0-24-generic",
"family": "debian"
},
"id": "26b1ceca54c197067320f0bf634024c6"
},
"service": {
"type": "auditd"
},
"process": {
"executable": "/usr/lib/sm.bin/sendmail",
"name": "sendmail-mta",
"pid": 21089,
"ppid": 2387
},
"my_newfield": "10.0.2.24",
"destination": {
"port": "53",
"ip": "10.0.2.24"
}
},
"fields": {
"@timestamp": [
"2023-07-31T14:06:12.128Z"
]
},
"highlight": {
"event.module": [
"@kibana-highlighted-field@auditd@/kibana-highlighted-field@"
]
},
"sort": [
1690812372128
]
}
```
---
### Author:  [@pero](https://discuss.elastic.co/u/pero)
#### Post date: [July 31, 2023, 2:17pm UTC](https://discuss.elastic.co/t/logstash-filter-mutate-problem/339690/11 "2023-07-31T14:17:44Z")
As you can see there are two json document and I need to pass the field of one document into another document. Or possibly can I combine both documents to produce a new document? and if yes how do I do that.
Thanks for your swift response
Regards
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [July 31, 2023, 2:55pm UTC](https://discuss.elastic.co/t/logstash-filter-mutate-problem/339690/12 "2023-07-31T14:55:45Z")
> [@pero](#):
>
> As you can see there are two json document and I need to pass the field of one document into another document.
Then this is also expected, Logstash events are independent from each other, so if you created a field on one document, it will only exists on that document, you can't use this field on other events.
What you can try to do is to use the `aggregate` filter to combine two events and create one document, you need to have a unique identificar in both events to be able to combine them.
> [@pero](#):
>
> Or possibly can I combine both documents to produce a new document? and if yes how do I do that.
As mentioned before, you can try to use the [aggregate](https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html) filter to combine two events, you would need to have a unique identificar in both events to combine them and you will also need to run the pipeline with just **one** worker.
I do not use the `aggregate` filter so I can not help further, but there are a couple of messages in the forum about it that may help your use case.
I suggest you open a new topic as this is a completely different issue.
---
### Author:  [@pero](https://discuss.elastic.co/u/pero)
#### Post date: [July 31, 2023, 3:01pm UTC](https://discuss.elastic.co/t/logstash-filter-mutate-problem/339690/13 "2023-07-31T15:01:14Z")
thanks
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 28, 2023, 3:01pm UTC](https://discuss.elastic.co/t/logstash-filter-mutate-problem/339690/14 "2023-08-28T15:01:24Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.