# Mutate a Nested Field

**URL:** <https://discuss.elastic.co/t/mutate-a-nested-field/231954>\
**Category:** Logstash\
**Tags:** elastic-stack-monitoring\
**Created:** [May 11, 2020, 8:27am UTC](https://discuss.elastic.co/t/mutate-a-nested-field/231954 "2020-05-11T08:27:10Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![ckough](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ckough/32/61892_2.png) [@ckough](https://discuss.elastic.co/u/ckough)\
**Post date:** [May 11, 2020, 8:27am UTC](https://discuss.elastic.co/t/mutate-a-nested-field/231954/1 "2020-05-11T08:27:10Z")

</div>

Hi there,

I have an index that looks like so:

```auto
{
  "_index": "ec2_logs-index-2020.05.11",
  "_type": "_doc",
  "_id": "V3LAAnIBk2krYRHTuxCC",
  "_version": 1,
  "_score": null,
  "_source": {
    "owner": "104115521938",
    "@version": "1",
    "logStream": "mystream",
    "subscriptionFilters": [
      "To-Firehose"
    ],
    "messageType": "DATA_MESSAGE",
    "logGroup": "ec2-fluentd",
    "@timestamp": "2020-05-11T08:01:24.400Z",
    "logEvents": [
      {
        "timestamp": 1589167148000,
        "id": "35439611645423604170418303021688828486859711444132757504",
        "message": "{\"host\":\"ip-22-22-2-83\",\"ident\":\"dhclient\",\"pid\":\"2987\",\"message\":\"XMT: Solicit on eth0, interval 109630ms.\"}"
      },
      {
        "timestamp": 1589167161000,
        "id": "35439611935333291751316403861653166031288411021878165505",
        "message": "{\"host\":\"ip-22-22-2-83\",\"ident\":\"sshd\",\"pid\":\"3864\",\"message\":\"Accepted publickey for ec2-user from 138.75.33.59 port 51868 ssh2: RSA SHA256:QY/2uJJiV5cYpErAb/KLg/505B6WQ4ZlvcBazh1Qfyo\"}"
      },
      {
        "timestamp": 1589167161000,
        "id": "35439611935333291751316403861653166031288411021878165506",
        "message": "{\"host\":\"ip-22-22-2-83\",\"ident\":\"systemd\",\"message\":\"Created slice User Slice of ec2-user.\"}"
      },
      {
        "timestamp": 1589167161000,
        "id": "35439611935333291751316403861653166031288411021878165507",
        "message": "{\"host\":\"ip-22-22-2-83\",\"ident\":\"systemd\",\"message\":\"Starting User Slice of ec2-user.\"}"
      },
      {
        "timestamp": 1589167161000,
        "id": "35439611935333291751316403861653166031288411021878165508",
        "message": "{\"host\":\"ip-22-22-2-83\",\"ident\":\"systemd\",\"message\":\"Started Session 2 of user ec2-user.\"}"
      },
      {
        "timestamp": 1589167161000,
        "id": "35439611935333291751316403861653166031288411021878165509",
        "message": "{\"host\":\"ip-22-22-2-83\",\"ident\":\"systemd-logind\",\"message\":\"New session 2 of user ec2-user.\"}"
      }
    ],
    "type": "ec2_logs"
  },
  "fields": {
    "@timestamp": [
      "2020-05-11T08:01:24.400Z"
    ]
  },
  "sort": [
    1589184084400
  ]
}

```

The index mapping is:

```auto
{
  "ec2_logs-index-2020.05.11" : {
    "mappings" : {
      "dynamic" : "true",
      "_meta" : { },
      "_source" : {
        "includes" : [],
        "excludes" : []
      },
      "dynamic_date_formats" : [
        "strict_date_optional_time",
        "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
      ],
      "dynamic_templates" : [],
      "date_detection" : true,
      "numeric_detection" : false,
      "properties" : {
        "@timestamp" : {
          "type" : "date",
          "format" : "strict_date_optional_time"
        },
        "@version" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "logEvents" : {
          "type" : "nested",
          "properties" : {
            "id" : {
              "type" : "text"
            },
            "message" : {
              "type" : "text"
            },
            "timestamp" : {
              "type" : "date_nanos"
            }
          }
        },
        "logGroup" : {
          "type" : "text"
        },
        "logStream" : {
          "type" : "text"
        },
        "messageType" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "owner" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "subscriptionFilters" : {
          "type" : "text"
        },
        "type" : {
          "type" : "text"
        }
      }
    }
  }
}

```

What I'm trying to do here is removing all forward slash and square brackets from logEvents.messages field and change logEvents.id to just a -

To that end, I have the following filter in logstash:

```auto
filter
{
  mutate{
    gsub => [
      "[logEvents][message]", "[\[\]\\]", "",
      "[logEvents][id]", ".*", "-"
    ]
  }
}

```

But the resulting output is exactly the same. What did i do wrong?

Thanks in advance...  
ck

---

<div class="post-metadata">

**Author:** ![andres-perez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andres-perez/32/136461_2.png) [@andres-perez](https://discuss.elastic.co/u/andres-perez)\
**Post date:** [May 11, 2020, 8:55pm UTC](https://discuss.elastic.co/t/mutate-a-nested-field/231954/2 "2020-05-11T20:55:32Z")

</div>

Hi. Your problem is that you actually want to iterate over an _array of objects with fields_ instead of a single, nested field.

Unfortunately, [gsub only works with strings or arrays of strings](https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-gsub).

Instead of

> [@ckough](#):
>
> ```
> gsub => [
> "[logEvents][message]", "[\[\]\\]", "",
> 
> ```

it should be something like

```auto
    gsub => [
          "[logEvents][0][message]", "[\[\]\\]", "",
          "[logEvents][1][message]", "[\[\]\\]", "",
          etc.

```

so, for a variable number of _LogEvents_ elements, I guess you will need to use a custom ruby filter to read _logEvents_ array and apply a substitution to all the _message_ values

---

<div class="post-metadata">

**Author:** ![ckough](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ckough/32/61892_2.png) [@ckough](https://discuss.elastic.co/u/ckough)\
**Post date:** [May 12, 2020, 2:07am UTC](https://discuss.elastic.co/t/mutate-a-nested-field/231954/3 "2020-05-12T02:07:37Z")

</div>

Hi Andres,

Yes you are right. There is an indeterminate number of logEvents elements. Sorry I have never done any ruby scripting before. I have googled a piece of code that'll probably return me the number of elements.

```auto
ruby {
  code => "
    event.set('number_of_elements', event.get('logEvents').length)
  "
}

```

After some googling, probably (?) the ruby code is:

```auto
ruby {
  code => "
    event.set('number_of_elements', event.get('logEvents').length)
    count = 0
    while count <= number_of_elements do
      mutate {   
        gsub => [
          "[logEvents][count][message]", "[\[\]\\]", "",
        ]
      }
    count +=1
    end
  "
}

```

Does it work like that?

thanks,  
ck

---

<div class="post-metadata">

**Author:** ![andres-perez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andres-perez/32/136461_2.png) [@andres-perez](https://discuss.elastic.co/u/andres-perez)\
**Post date:** [May 12, 2020, 11:19am UTC](https://discuss.elastic.co/t/mutate-a-nested-field/231954/4 "2020-05-12T11:19:12Z")

</div>

I presume you will have easier test environment to check your own data 🤨 and unfortunately I have just the bare minimum ruby knowledge to fulfill my needs regarding to elastic stack / logstash usage.

So I can ony give you general advice here, and you must take it as a non-rigorous descriptions at best 😃

- The _mutate_ block inside a _while_ loop is "logstash configuration/code", not Ruby. You can not mix both inside a ruby block.

- There is a ruby gsub method that you can apply to strings.

- The while loop may be improved, but I'll show another quick&dirty solution that may serve as inspiration:

```auto
    code => "
        event.set("logEvents", event.get("logEvents").each{ |item| item[:message].gsub!( /[\[\]\\]/, "" ) } )
    "

```

Notes:

- I haven't seen brackets or _back_ slashes (other than the ones used to escape json quotes etc) `[, [, \` in your _message_ fields, you will need to pick other examples to test it.

- Maybe you will need a more robust solution: check that logevents is not empty by assigning it to a intermediate variable and adding conditions, etc.

- More experienced ruby devs may give better insights or a more efficient, elegant, etc. solution.

---

<div class="post-metadata">

**Author:** ![Fabio-sama](https://avatars.discourse-cdn.com/v4/letter/f/b9e5f3/32.png) [@Fabio-sama](https://discuss.elastic.co/u/Fabio-sama)\
**Post date:** [May 12, 2020, 1:20pm UTC](https://discuss.elastic.co/t/mutate-a-nested-field/231954/5 "2020-05-12T13:20:33Z")

</div>

Hi there,

something like the following should work in your case:

```
filter {
  ruby {
    code => "
      log_events = event.get('logEvents')
      log_events.map do |event|
        event['id'] = '-'
        event['message'].gsub!(/[\[\]\/]/, '')
      end
      event.set('logEvents', log_events)
    "
  }
}

```

I tested it with the following pipeline (giving in input the sample you posted) and it seems to work properly:

```
input {
  generator { 
    count => 1
    lines => ['{"owner": "104115521938","@version": "1","logStream": "mystream","subscriptionFilters": ["To-Firehose"],"messageType": "DATA_MESSAGE","logGroup": "ec2-fluentd","@timestamp": "2020-05-11T08:01:24.400Z","logEvents": [{"timestamp": 1589167148000,"id": "35439611645423604170418303021688828486859711444132757504","message": "{\"host\":\"ip-22-22-2-83\",\"ident\":\"dhclient\",\"pid\":\"2987\",\"message\":\"XMT: Solicit on eth0, interval 109630ms.\"}"},{"timestamp": 1589167161000,"id": "35439611935333291751316403861653166031288411021878165505","message": "{\"host\":\"ip-22-22-2-83\",\"ident\":\"sshd\",\"pid\":\"3864\",\"message\":\"Accepted publickey for ec2-user from 138.75.33.59 port 51868 ssh2: RSA SHA256:QY/2uJJiV5cYpErAb/KLg/505B6WQ4ZlvcBazh1Qfyo\"}"},{"timestamp": 1589167161000,"id": "35439611935333291751316403861653166031288411021878165506","message": "{\"host\":\"ip-22-22-2-83\",\"ident\":\"systemd\",\"message\":\"Created slice User Slice of ec2-user.\"}"},{"timestamp": 1589167161000,"id": "35439611935333291751316403861653166031288411021878165507","message": "{\"host\":\"ip-22-22-2-83\",\"ident\":\"systemd\",\"message\":\"Starting User Slice of ec2-user.\"}"},{"timestamp": 1589167161000,"id": "35439611935333291751316403861653166031288411021878165508","message": "{\"host\":\"ip-22-22-2-83\",\"ident\":\"systemd\",\"message\":\"Started Session 2 of user ec2-user.\"}"},{"timestamp": 1589167161000,"id": "35439611935333291751316403861653166031288411021878165509","message": "{\"host\":\"ip-22-22-2-83\",\"ident\":\"systemd-logind\",\"message\":\"New session 2 of user ec2-user.\"}"}],"type": "ec2_logs"}' ]
    codec => "json"
  }
}

filter {
  ruby {
    code => "
      log_events = event.get('logEvents')
      log_events.map do |event|
        event['id'] = '-'
        event['message'].gsub!(/[\[\]\/]/, '')
      end
      event.set('logEvents', log_events)
    "
  }
}

output {
  stdout{}
}

```

However, I set the `id` nested field as the string `-`. If you want it to be `nil` just replace the `-` hardcoded string with `nil` obviously.

---

<div class="post-metadata">

**Author:** ![ckough](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ckough/32/61892_2.png) [@ckough](https://discuss.elastic.co/u/ckough)\
**Post date:** [May 15, 2020, 7:28am UTC](https://discuss.elastic.co/t/mutate-a-nested-field/231954/6 "2020-05-15T07:28:19Z")

</div>

Thanks Fabio and Andres! 👍 This is exactly what I'm looking for.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 12, 2020, 7:28am UTC](https://discuss.elastic.co/t/mutate-a-nested-field/231954/7 "2020-06-12T07:28:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.