# Move subarrays to document root

**URL:** <https://discuss.elastic.co/t/move-subarrays-to-document-root/143876>\
**Category:** Logstash\
**Created:** [August 10, 2018, 1:49pm UTC](https://discuss.elastic.co/t/move-subarrays-to-document-root/143876 "2018-08-10T13:49:20Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![programagor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/programagor/32/40183_2.png) [@programagor](https://discuss.elastic.co/u/programagor)\
**Post date:** [August 10, 2018, 1:49pm UTC](https://discuss.elastic.co/t/move-subarrays-to-document-root/143876/1 "2018-08-10T13:49:20Z")

</div>

Greetings  
My incoming messages contain subarrays in the following format:

```
com.openexchange.ajax.action
com.openexchange.ajax.module
com.openexchange.database.schema
com.openexchange.grizzly.method
com.openexchange.grizzly.queryString
com.openexchange.grizzly.remoteAddress
com.openexchange.grizzly.remotePort
com.openexchange.grizzly.requestURI
com.openexchange.grizzly.serverName
com.openexchange.grizzly.servletPath
com.openexchange.grizzly.session
com.openexchange.grizzly.threadName
com.openexchange.grizzly.userAgent
com.openexchange.login.authId
com.openexchange.login.client
com.openexchange.login.clientIp
com.openexchange.login.login
com.openexchange.login.userAgent
com.openexchange.login.version
com.openexchange.mail.accountId
com.openexchange.mail.host
com.openexchange.mail.login
com.openexchange.mail.session
com.openexchange.request.trackingId
com.openexchange.session.authId
com.openexchange.session.clientId
com.openexchange.session.contextId
com.openexchange.session.sessionId
com.openexchange.session.userId
com.openexchange.session.userName

```

However, I would like to strip off the `com.openexchange` part of each field. Is that possible? Preferrably without having to list the `ajax`, `database`, etc. subfields.  
I tried this filter, but it doesn't move anything:

```
mutate {
    rename => { "com.openexchange.grizzly" => "grizzly" }
    rename => { "com.openexchange.mail" => "mail" }
    rename => { "com.openexchange.request" => "request" }
    rename => { "com.openexchange.database"=> "database" }
    rename => { "com.openexchange.ajax" => "ajax" }
    rename => { "com.openexchange.login" => "login" }
    rename => { "com.openexchange.session" => "session" }
}

```

I also tried a Ruby filter, but it creates fields with null in them:

```
ruby {
    code => '
        event.set("[grizzly]", event.get("[com][openexchange][grizzly]"))
        event.set("[mail]", event.get("[com][openexchange][mail]"))
        event.set("[request]", event.get("[com][openexchange][request]"))
        event.set("[database]", event.get("[com][openexchange][database]"))
        event.set("[ajax]", event.get("[com][openexchange][ajax]"))
        event.set("[login]", event.get("[com][openexchange][login]"))
        event.set("[session]", event.get("[com][openexchange][session]"))
    '
}

```

Of course, renaming individual leafs will work, but that's tedious, and not general in case I add more in the log source.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 10, 2018, 2:03pm UTC](https://discuss.elastic.co/t/move-subarrays-to-document-root/143876/2 "2018-08-10T14:03:24Z")

</div>

```
    ruby { code => 'event.get("[com][openexchange]").each { | k, v| event.set(k, v) }' }
    mutate { remove_field => ["[com][openexchange]", "com" ] }
```

---

<div class="post-metadata">

**Author:** ![programagor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/programagor/32/40183_2.png) [@programagor](https://discuss.elastic.co/u/programagor)\
**Post date:** [August 10, 2018, 2:29pm UTC](https://discuss.elastic.co/t/move-subarrays-to-document-root/143876/3 "2018-08-10T14:29:30Z")

</div>

Thanks, but that didn't do it. Now I'm getting

```
Ruby exception occurred: undefined method `each' for nil:NilClass

```

However, the fields should exist, I certainly do see them in Kibana.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 10, 2018, 3:45pm UTC](https://discuss.elastic.co/t/move-subarrays-to-document-root/143876/4 "2018-08-10T15:45:45Z")

</div>

> [@programagor](#):
>
> However, the fields should exist, I certainly do see them in Kibana.

What **exactly** do you see in Kibana. Are they arrays? Can you copy and paste from the JSON tab?

---

<div class="post-metadata">

**Author:** ![programagor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/programagor/32/40183_2.png) [@programagor](https://discuss.elastic.co/u/programagor)\
**Post date:** [August 10, 2018, 4:17pm UTC](https://discuss.elastic.co/t/move-subarrays-to-document-root/143876/5 "2018-08-10T16:17:28Z")

</div>

Here is a part of one document that I can see in Kibana:

```
{
  "_index": "clog-mail-2018.08.10",
  "_type": "doc",
  "_id": "SqKWJGUBcUjz8ltKlwa6",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2018-08-10T16:06:08.891Z",
    "@version": 1,
    "com.openexchange.grizzly.method": "GET",
    "com.openexchange.grizzly.threadName": "OXWorker-0001932",
    "com.openexchange.grizzly.remotePort": "18716",
    ...
    "tags": [
      "oxbe",
      "open-xchange",
      "beats_input_codec_plain_applied",
      "_rubyexception",
    ]
  },
  "fields": {
    "@timestamp": [
      "2018-08-10T16:06:08.891Z"
    ]
  }
}

```

So, my mistake, they don't seem to be actual arrays. I apologise for the confusion.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 10, 2018, 4:34pm UTC](https://discuss.elastic.co/t/move-subarrays-to-document-root/143876/6 "2018-08-10T16:34:18Z")

</div>

OK, so you have dots in your field names. That is unsupported, and I know there are things that do not work with such fields.

Take a look at the [de\_dot](https://www.elastic.co/blog/introducing-the-de_dot-filter) filter. Note that it explicitly warns that it is expensive to run. But this would transform that into something that the ruby filter would work on.

```
de_dot { nested => true }
```

---

<div class="post-metadata">

**Author:** ![programagor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/programagor/32/40183_2.png) [@programagor](https://discuss.elastic.co/u/programagor)\
**Post date:** [August 13, 2018, 2:35pm UTC](https://discuss.elastic.co/t/move-subarrays-to-document-root/143876/7 "2018-08-13T14:35:33Z")

</div>

Excellent, that works! However, there are some clashes with fields that already exist in the document (e.g. `requests` already exists as a string). What would be the easiest way to store all of the subfields into an `ox` field rather than the document root?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 13, 2018, 3:06pm UTC](https://discuss.elastic.co/t/move-subarrays-to-document-root/143876/8 "2018-08-13T15:06:03Z")

</div>

A slight change to the ruby filter

```
ruby { code => 'event.get("[com][openexchange]").each { | k, v| event.set("[ox][" + k + "]", v) }' }
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 10, 2018, 3:06pm UTC](https://discuss.elastic.co/t/move-subarrays-to-document-root/143876/9 "2018-09-10T15:06:08Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.