# Filter to add fiels of on nested JSON name value fields

**URL:** <https://discuss.elastic.co/t/filter-to-add-fiels-of-on-nested-json-name-value-fields/300848>\
**Category:** Logstash\
**Created:** [March 28, 2022, 1:24pm UTC](https://discuss.elastic.co/t/filter-to-add-fiels-of-on-nested-json-name-value-fields/300848 "2022-03-28T13:24:29Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![DI\_Ralph](https://avatars.discourse-cdn.com/v4/letter/d/f0a364/32.png) [@DI\_Ralph](https://discuss.elastic.co/u/DI_Ralph)\
**Post date:** [March 28, 2022, 1:24pm UTC](https://discuss.elastic.co/t/filter-to-add-fiels-of-on-nested-json-name-value-fields/300848/1 "2022-03-28T13:24:29Z")

</div>

Hi,  
In Logstash I'm trying to create new fields from Nested JSON where the value of "Name" is part of the field name and "Value" is the actual value of the new field.

For example I got part of an audit log here below.

```auto
data.DeviceProperties	
{
  "Name": "OS",
  "Value": "Windows 10"
},
{
  "Name": "BrowserType",
  "Value": "Firefox"
},
{
  "Name": "IsCompliantAndManaged",
  "Value": "False"
}

```

Now from this I wish to see 3 new fields being present in Kibana, namely:  
data.DeviceProperties.OS: "Windows 10"  
data.DeviceProperties.BrowserType: "FireFox"  
data.DeviceProperties.IsCompliantAndManaged: False

I hope someone has an solution or suggestion how I can achieve this in Logstash.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 28, 2022, 2:54pm UTC](https://discuss.elastic.co/t/filter-to-add-fiels-of-on-nested-json-name-value-fields/300848/2 "2022-03-28T14:54:23Z")

</div>

You could do something like [this](https://discuss.elastic.co/t/how-can-i-flatten-an-array-in-my-document/235807/3).

---

<div class="post-metadata">

**Author:** ![DI\_Ralph](https://avatars.discourse-cdn.com/v4/letter/d/f0a364/32.png) [@DI\_Ralph](https://discuss.elastic.co/u/DI_Ralph)\
**Post date:** [March 29, 2022, 7:37am UTC](https://discuss.elastic.co/t/filter-to-add-fiels-of-on-nested-json-name-value-fields/300848/3 "2022-03-29T07:37:39Z")

</div>

Thank you, it looks like it might be the thing I was looking for. I'll go try it and will let you know if it worked.

---

<div class="post-metadata">

**Author:** ![DI\_Ralph](https://avatars.discourse-cdn.com/v4/letter/d/f0a364/32.png) [@DI\_Ralph](https://discuss.elastic.co/u/DI_Ralph)\
**Post date:** [March 30, 2022, 9:18am UTC](https://discuss.elastic.co/t/filter-to-add-fiels-of-on-nested-json-name-value-fields/300848/4 "2022-03-30T09:18:31Z")

</div>

Hi, I'm getting an unexpected error I can't seem to resolve.

```auto
[2022-03-30T10:58:44,651][ERROR][logstash.codecs.json][beats-server][219d0691fbdb75c864a5b62548deb7f17646341a375b877ee0e35512e7ef3280] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected end-of-input within/between Object entries
 at [Source: (String)"{"CreationTime": "2021-11-08T10:10:14", "Id": "x", "Operation": "UserLoggedIn", "OrganizationId": "x", "RecordType": 15, "ResultStatus": "Success", "UserKey": "x", "UserType": 0, "Version": 1, "Workload": "AzureActiveDirectory", "ClientIP": "x", "ObjectId": "x", "UserId": "x", "AzureActiveDirectoryEventType": "[truncated 24 chars]; line: 1, column: 1049]>, :data=>"{\"CreationTime\": \"2021-11-08T10:10:14\", \"Id\": \"x\", \"Operation\": \"UserLoggedIn\", \"OrganizationId\": \"x\", \"RecordType\": 15, \"ResultStatus\": \"Success\", \"UserKey\": \"x\", \"UserType\": 0, \"Version\": 1, \"Workload\": \"AzureActiveDirectory\", \"ClientIP\": \"x\", \"ObjectId\": \"x\", \"UserId\": \"x\", \"AzureActiveDirectoryEventType\": 1, \"ExtendedProperties\":"}

```

The full log it is trying to parse:

```auto
{"CreationTime": "2021-11-08T10:10:14", "Id": "x", "Operation": "UserLoggedIn", "OrganizationId": "x", "RecordType": 15, "ResultStatus": "Success", "UserKey": "x", "UserType": 0, "Version": 1, "Workload": "AzureActiveDirectory", "ClientIP": "x", "ObjectId": "x", "UserId": "x", "AzureActiveDirectoryEventType": 1, "ExtendedProperties":
[{"Name": "ResultStatusDetail", "Value": "Success"}, {"Name": "UserAgent", "Value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.0 Safari/605.1.15"}, {"Name": "RequestType", "Value": "OAuth2:Authorize"}], "ModifiedProperties": [], "Actor": [{"ID": "x", "Type": 0}, {"ID": "x", "Type": 5}], "ActorContextId": "x", "ActorIpAddress": "x", "InterSystemsId": "x", "IntraSystemId": "x", "SupportTicketId": "", "Target": [{"ID": "x", "Type": 0}], "TargetContextId": "x", "ApplicationId": "x", "DeviceProperties": [{"Name": "OS", "Value": "MacOs"}, {"Name": "BrowserType", "Value": "Safari"}, {"Name": "IsCompliantAndManaged", "Value": "False"}, {"Name": "SessionId", "Value": "x"}], "ErrorNumber": "0", "customerName": "x"}

```

The content of the logstash.conf as of now:

```auto
input {
  pipeline { address => "office365" }
}

filter {
   json {
     source => "DeviceProperties"
   }
       ruby {
        code => '
        props = event.get("DeviceProperties")
        if props
            props.each { |x|
                key = x["key"]
                event.set("DeviceProperties.#{key}", x["value"])
            }
            event.remove("DeviceProperties")
        end
    '
    }
}

output {
  elasticsearch {    
  }
}

```

And pipelines.yml

```auto
- pipeline.id: beats-server
  config.string:
    input {
      beats {
        port => 5046
        codec => "json"
      }
    }

    output {
      pipeline {
        send_to => "office365"
      }
    }

- pipeline.id: office365
  path.config: "/etc/logstash/conf.d/logstash.conf"

```

What I've tried this far is changing the input codec from "json" to "json\_lines" and I've tried it without specifying the "json source" in the filter but unfortunaly to no prevail.

Hope you can see what I'm doing wrong here!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 30, 2022, 6:26pm UTC](https://discuss.elastic.co/t/filter-to-add-fiels-of-on-nested-json-name-value-fields/300848/5 "2022-03-30T18:26:40Z")

</div>

> [@DI\_Ralph](#):
>
> The full log it is trying to parse:

That is two lines. You are going to need to combine them before using a json filter or codec. You will probably want to solve that upstream in the beat.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 27, 2022, 6:27pm UTC](https://discuss.elastic.co/t/filter-to-add-fiels-of-on-nested-json-name-value-fields/300848/6 "2022-04-27T18:27:10Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.