# Flatten json and create dynamic field

**URL:** <https://discuss.elastic.co/t/flatten-json-and-create-dynamic-field/240360>\
**Category:** Logstash\
**Created:** [July 8, 2020, 1:57pm UTC](https://discuss.elastic.co/t/flatten-json-and-create-dynamic-field/240360 "2020-07-08T13:57:14Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [July 8, 2020, 1:57pm UTC](https://discuss.elastic.co/t/flatten-json-and-create-dynamic-field/240360/1 "2020-07-08T13:57:14Z")

</div>

I know ruby will be involve I have try few code that I found on forum. but didn't work out

here is my input

message =\> '{"jobid":348332180, "runtime":{"cputime": 24, "totalelapsed": 105},"dataflow":[{"gate": "v204.06", "attrmath":"v206.07","keep": "lib"}]}'

dataflow will have different name different value for each message. upto 150 different name  
like gate, attrmath, keep, in, out etc..

would like to create document something like this, means creating field dynamically what ever appears in dataflow array.

```
{
    jobid:348332180
    runtime_cputime:24
    runtime_totalelapsed:105
   gate_name: gate
   gate_version: v204.06
   attrmath_name: attrmath
   attrmath_version: v206.07
   keep_name: keep
   keep_version: lib
}

```

in end I might have 200 field in my index but I think better then nested structure.

---

<div class="post-metadata">

**Author:** ![grumo35](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grumo35/32/59451_2.png) [@grumo35](https://discuss.elastic.co/u/grumo35)\
**Post date:** [July 8, 2020, 2:23pm UTC](https://discuss.elastic.co/t/flatten-json-and-create-dynamic-field/240360/2 "2020-07-08T14:23:29Z")

</div>

Hey,

Will the json plugin do the job ?

I'm not sure if understand the output you wanted but you should try something like

```auto
filter{ json { source => "message" } }

```

And the field will be accessible trough [jobid] or [runtime][cputime]

If you want to extract field and just add the content without nested fields just rename them.

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [July 8, 2020, 2:27pm UTC](https://discuss.elastic.co/t/flatten-json-and-create-dynamic-field/240360/3 "2020-07-08T14:27:46Z")

</div>

all that is been done. but output is like this

```
{
      "dataflow" => [
        [0] {
                "keep" => "lib",
                "gate" => "v204.06",
            "attrmath" => "v206.07"
        }
    ],
      "@version" => "1",
    "@timestamp" => 2020-07-08T14:25:58.605Z,
                "jobid" => 348332180,
       "runtime" => {
             "cputime" => 24,
        "totalelapsed" => 105
    }
}

```

but I want this dataflow array to be flatten out.

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [July 8, 2020, 3:20pm UTC](https://discuss.elastic.co/t/flatten-json-and-create-dynamic-field/240360/4 "2020-07-08T15:20:30Z")

</div>

used this code but didn't work

```
if [dataflow] {
   ruby {
     code => '
         kv = event.get("dataflow")
          kv.to_hash.each { |k,v|
         event.set(k, v)
      }
   '
   }
}
```

---

<div class="post-metadata">

**Author:** ![grumo35](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grumo35/32/59451_2.png) [@grumo35](https://discuss.elastic.co/u/grumo35)\
**Post date:** [July 8, 2020, 3:51pm UTC](https://discuss.elastic.co/t/flatten-json-and-create-dynamic-field/240360/5 "2020-07-08T15:51:45Z")

</div>

Have you tried to KV from logstash ?

You can play with fields and logstash

```auto
filter{
  json { source => "message" 
  target => "json"
  } 
}
# changing nested fields into top level fields
mutate{
rename =>{"[json][jobid]" => "jobid"}
rename =>{"[json][runtime]" => "runtime"}
}
# Extracting double nested field for kv
kv {
  source => [json][dataflow]
}
mutate{ remove_field => ["json"] }

```

This will be pretty ok i assume ?

Can i ask you why it is a problem to keep nested fields ?

---

<div class="post-metadata">

**Author:** ![grumo35](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grumo35/32/59451_2.png) [@grumo35](https://discuss.elastic.co/u/grumo35)\
**Post date:** [July 8, 2020, 3:54pm UTC](https://discuss.elastic.co/t/flatten-json-and-create-dynamic-field/240360/6 "2020-07-08T15:54:40Z")

</div>

You'll have to find a way to dynamically rename unknown fields ?

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [July 8, 2020, 3:54pm UTC](https://discuss.elastic.co/t/flatten-json-and-create-dynamic-field/240360/7 "2020-07-08T15:54:41Z")

</div>

this index can go with many million record and don't want nested field as it will cause more problem then not

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [July 8, 2020, 3:55pm UTC](https://discuss.elastic.co/t/flatten-json-and-create-dynamic-field/240360/8 "2020-07-08T15:55:12Z")

</div>

hope I will wait for someone to give some path on ruby code for this

---

<div class="post-metadata">

**Author:** ![Jenni](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jenni/32/29684_2.png) [@Jenni](https://discuss.elastic.co/u/Jenni)\
**Post date:** [July 8, 2020, 4:18pm UTC](https://discuss.elastic.co/t/flatten-json-and-create-dynamic-field/240360/9 "2020-07-08T16:18:31Z")

</div>

> [@elasticforme](#):
>
> used this code but didn't work
> 
> ```auto
> if [dataflow] {
> ruby {
> code => '
> kv = event.get("dataflow")
> kv.to_hash.each { |k,v|
> event.set(k, v)
> }
> '
> }
> }
> 
> ```

With "flatten out" you mean transfer those fields to the root level of your event, right? Then your code is close to the solution. But dataflow is an array with the actual data in the first entry. And you need to remove the field afterwards. I think it should be like this:

```auto
if [dataflow] {
   ruby {
     code => '
         event.get("[dataflow][0]").each { |k,v|
           event.set(k, v)
         }
         event.remove("dataflow")
   '
   }
}

```

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [July 8, 2020, 5:03pm UTC](https://discuss.elastic.co/t/flatten-json-and-create-dynamic-field/240360/10 "2020-07-08T17:03:23Z")

</div>

Very good Thank you it is working

final code

```
if [dataflow] {
   ruby {
     code => '
         event.get("[dataflow][0]").each { |k,v|
           event.set("#{k}_name", k)
           event.set("#{k}_version", v)
         }
         event.remove("dataflow")
   '
   }
}
if [runtime] {
   ruby {
      code => '
          event.get("[runtime]").each { |k,v|
          event.set(k,v)
          }
          event.remove("runtime")
      '
    }
}

```

output something like this

```
   "gate_name" => "gate",
   "gate_version" => "v204.06",
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 5, 2020, 5:03pm UTC](https://discuss.elastic.co/t/flatten-json-and-create-dynamic-field/240360/11 "2020-08-05T17:03:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.