# Logstash: manipulate json data

**URL:** <https://discuss.elastic.co/t/logstash-manipulate-json-data/254198>\
**Category:** Logstash\
**Created:** [November 4, 2020, 3:06am UTC](https://discuss.elastic.co/t/logstash-manipulate-json-data/254198 "2020-11-04T03:06:19Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![nameisnotimportant](https://avatars.discourse-cdn.com/v4/letter/n/ea5d25/32.png) [@nameisnotimportant](https://discuss.elastic.co/u/nameisnotimportant)\
**Post date:** [November 4, 2020, 3:06am UTC](https://discuss.elastic.co/t/logstash-manipulate-json-data/254198/1 "2020-11-04T03:06:19Z")

</div>

Hi,

i retrieved data from ES and would like to add/modify the json data before index it.

My json data looks like below:

> ```
> {
> "john" : {
> "enabled" : true,
> "roles" : [
> "developer"
> ],
> "rules" : {
> "any" : [
> {
> "field" : {
> "dn" : "CN=john,OU=IT,OU=MY,DC=DomainName,DC=local"
> }
> }
> ]
> },
> "metadata" : { }
> },
> "shyap" : {
> "enabled" : true,
> "roles" : [
> "superuser",
> "reportinguser"
> ],
> "rules" : {
> "any" : [
> {
> "field" : {
> "dn" : "CN=shyap,OU=IT,OU=MY,DC=DomainName,DC=local"
> }
> }
> ]
> },
> "metadata" : { }
> }
> }
> 
> ```

My config file:

> ```
> input {
> http_poller {
> urls => {
> test1 => { 
> method => get
> user => "elastic"
> password => "changeme"
> url => "https://hostname:9200/_security/role_mapping"
> headers => {
> Accept => "application/json"
> }
> }
> }    
> request_timeout => 60
> schedule => {"every" => "2s"}
> codec => "json"
> cacert => "/etc/elasticsearch/certs/root2016.crt"
> }
> }
> filter {
> json {
> source => "message"
> }
> mutate {
> remove_field => ["@timestamp"]
> remove_field => ["@version"]
> remove_field => ["message"]
> }
> }
> output {
> stdout { codec => rubydebug }
> }
> 
> ```

This is the role mapping data.

I would like to know how do i transform the data into following:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/0/70bf5566642a761d5e1c0d15fbccc11033144181.png)

First step i am trying is to add new field 'userid' and give the value of 'shyap' and 'john'.

When i put in the 'add\_field' option, nothing happne.

i use following json filter:

> ```
> json {
> source => "message"
> add_field => {"userid" => "%{[message][0]}"}
> }
> 
> ```

However, if i use mutate filter, i get only one userid field added. Somehow, the two records were treated as one?

> ```
> {
> "shyap" => {
> "enabled" => true,
> "roles" => [
> [0] "superuser",
> [1] "reportinguser"
> ],
> "rules" => {
> "any" => [
> [0] {
> "field" => {
> "dn" => "CN=shyap,OU=IT,OU=MY,DC=DomainName,DC=local"
> }
> }
> ]
> },
> "metadata" => {}
> },
> "userid" => "%{[message][0]}",
> "john" => {
> "enabled" => true,
> "roles" => [
> [0] "developer"
> ],
> "rules" => {
> "any" => [
> [0] {
> "field" => {
> "dn" => "CN=john,OU=IT,OU=MY,DC=DomainName,DC=local"
> }
> }
> ]
> },
> "metadata" => {}
> }
> }
> 
> ```

Can someone guide me how do i proceed ?

Thanks in advance.

---

<div class="post-metadata">

**Author:** ![Wolfram\_Haussig](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wolfram_haussig/32/70528_2.png) [@Wolfram\_Haussig](https://discuss.elastic.co/u/Wolfram_Haussig)\
**Post date:** [November 4, 2020, 5:38am UTC](https://discuss.elastic.co/t/logstash-manipulate-json-data/254198/2 "2020-11-04T05:38:09Z")

</div>

Hi,

After parsing the JSON you have to use the [`split filter`](https://www.elastic.co/guide/en/logstash/current/plugins-filters-split.html). With this, you can split the single document into separate documents for each user. After this, your mutate filter should work.

Best regards  
Wolfram

---

<div class="post-metadata">

**Author:** ![nameisnotimportant](https://avatars.discourse-cdn.com/v4/letter/n/ea5d25/32.png) [@nameisnotimportant](https://discuss.elastic.co/u/nameisnotimportant)\
**Post date:** [November 4, 2020, 7:29am UTC](https://discuss.elastic.co/t/logstash-manipulate-json-data/254198/3 "2020-11-04T07:29:44Z")

</div>

i tried that with following:

> ```
> split {
> field => "message"
> }
> 
> ```

and i got error:

> [WARN] 2020-11-04 15:21:37.137 [[main]\>worker18] split - Only String and Array types are splittable. field:message is of type = NilClass

Either the json filter not working or the codec not working...i am yet to figure it out.

---

<div class="post-metadata">

**Author:** ![Wolfram\_Haussig](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wolfram_haussig/32/70528_2.png) [@Wolfram\_Haussig](https://discuss.elastic.co/u/Wolfram_Haussig)\
**Post date:** [November 4, 2020, 7:40am UTC](https://discuss.elastic.co/t/logstash-manipulate-json-data/254198/4 "2020-11-04T07:40:35Z")

</div>

I am sorry, I didn't see that the json is a hash instead of an array.

I fear that you have to convert that hash/dictionary to an array first using ruby. After that you can use the split filter to convert this array to separate documents.

Maybe another user has a better idea?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 4, 2020, 5:14pm UTC](https://discuss.elastic.co/t/logstash-manipulate-json-data/254198/5 "2020-11-04T17:14:04Z")

</div>

I would start with

```
filter {
    ruby {
        code => '
            a = []
            event.get("data").each { |k, v|
                a << v.merge( { "user" => k } )
            }
            event.set("users", a)
        '
        remove_field => ["data"]
    }
    split { field => "users" }
    mutate { join => { "[users][roles]" => "," } }
}

```

then you just need to move fields around using mutate.

---

<div class="post-metadata">

**Author:** ![nameisnotimportant](https://avatars.discourse-cdn.com/v4/letter/n/ea5d25/32.png) [@nameisnotimportant](https://discuss.elastic.co/u/nameisnotimportant)\
**Post date:** [November 5, 2020, 9:57am UTC](https://discuss.elastic.co/t/logstash-manipulate-json-data/254198/6 "2020-11-05T09:57:25Z")

</div>

Thank you Wolfram and Badger.  
With your help, i am almost done. What is pending now is the "dn" key value pair. I am still not able to put it at the same level as role.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/f/8f88613c64e2ecc7afd9810c94b1c5d7310da590.png)

This is my latest logstash config file:

> ```
> input {
> http_poller {
> urls => {
> test1 => { 
> method => get
> user => "elastic"
> password => "changeme"
> url => "https://hostname:9200/_security/role_mapping"
> headers => {
> Accept => "application/json"
> }
> }
> }    
> request_timeout => 60
> schedule => {"every" => "1s"}
> codec => "plain"
> cacert => "/etc/elasticsearch/certs/root2016.crt"
> }
> }
> 
> filter {
> ruby {
> code => '
> require "date"
> #add new field to apply to alias with other system
> a = []
> b = {"custom_string_field" => "nil"}
> c = {"custom_date_field" => DateTime.now.iso8601(3)}
> 
> JSON.parse(event.get("message")).each { |k, v|
> v.merge!(b)
> v.merge!(c)
> a << v.merge( { "userid" => k } )
> }
> event.set("users", a)
> '
> remove_field => ["message"]
> }
> 
> split {
> field => "users"
> }
> mutate {
> join => { "[users][roles]" => "," }
> }
> 
> mutate {
> remove_field => ["@timestamp"]
> remove_field => ["@version"]
> }
> 
> #to remove root
> ruby {
> code => '
> event.get("users").each { |k, v|
> event.set(k, v)
> }
> event.remove("users")
> 
> event.get("rules").each { |k, v|
> event.set(k, v)
> }
> event.remove("rules")
> 
> # the following commented part doesn't work
> #event.get("any").each { |k, v|
> # event.set(k, v)
> # }
> # event.remove("any")
> '
> }
> }
> 
> output {
> stdout { codec => rubydebug }
> }
> 
> ```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 5, 2020, 3:18pm UTC](https://discuss.elastic.co/t/logstash-manipulate-json-data/254198/7 "2020-11-05T15:18:05Z")

</div>

[any] is an array. You can move the dn to the root level using

```
mutate { rename => { "[any][0][field][dn]" => "dn" } }
```

---

<div class="post-metadata">

**Author:** ![nameisnotimportant](https://avatars.discourse-cdn.com/v4/letter/n/ea5d25/32.png) [@nameisnotimportant](https://discuss.elastic.co/u/nameisnotimportant)\
**Post date:** [November 6, 2020, 1:50am UTC](https://discuss.elastic.co/t/logstash-manipulate-json-data/254198/8 "2020-11-06T01:50:36Z")

</div>

Thanks Badger. That did the tricks !  
Although i still wonder around, what if i have more than one items in the array ? But, i prefer to try it out first before i ask for help.

Thanks a lot.  
Regards.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 4, 2020, 1:50am UTC](https://discuss.elastic.co/t/logstash-manipulate-json-data/254198/9 "2020-12-04T01:50:38Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.