# Logstash how to parse and split nested json file

**URL:** <https://discuss.elastic.co/t/logstash-how-to-parse-and-split-nested-json-file/305308>\
**Category:** Logstash\
**Created:** [May 20, 2022, 5:37pm UTC](https://discuss.elastic.co/t/logstash-how-to-parse-and-split-nested-json-file/305308 "2022-05-20T17:37:54Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Iss](https://avatars.discourse-cdn.com/v4/letter/i/58956e/32.png) [@Iss](https://discuss.elastic.co/u/Iss)\
**Post date:** [May 20, 2022, 5:37pm UTC](https://discuss.elastic.co/t/logstash-how-to-parse-and-split-nested-json-file/305308/1 "2022-05-20T17:37:54Z")

</div>

Hi!  
I have a nested json file in a list read as single event in Elasticsearch like this :

```auto
[{"header": {"id": "idvalue", 
			 "datestamp": "YYYY-MM-DD"
			 }, 
   "metas": {"dc:title": "text...", 
			 "dc:id": "idvalue",
			 "dc:subject": [{"id": "idvalue", "title": "text"}, 
							{"id": "idvalue", "title": "text"}, 
							{"id4": "idvalue", "title": "text..."}, 
							{"id": "idvalue", "title": "text"}
							], 
			 "dc:description": "long text.\r\n\r\n\r\n\r\n\r\n\r\n", 
			 "dc:pub": [{"id": "idvalue", "title": "text..."}
						], 
			 "dc:creation": "YYY-MM-DD:HH:mm:ss", 
			 "dc:modif": "YYY-MM-DD:HH:mm:ss", 
			 "dc:ava": "YYY-MM-DD:HH:mm:ss", 
			 "dc:typ": [{"id": "value", "title": "texte"}
						], 
			 "dc:ext": "HH:MM:SS", 
			 "dc:loc": [{"lat": numvalue, "lng": numvalue}
						], 
			"dc:lic": "text", 
			"dc:rH": [{"id": "value", "title": "text"}
						], 
			"dc:aud": "text"
			}
	}, 
	{"header": {"id": "idvalue", 
			 "datestamp": "YYYY-MM-DD"
			 }, 
   "metas": {"dc:title": "text...", 
			 "dc:id": "idvalue",
			 "dc:subject": [{"id": "idvalue", "title": "text"}, 
							{"id": "idvalue", "title": "text"}, 
							{"id4": "idvalue", "title": "text..."}, 
							{"id": "idvalue", "title": "text"}
							], 
			 "dc:description": "long text.\r\n\r\n\r\n\r\n\r\n\r\n", 
			 "dc:pub": [{"id": "idvalue", "title": "text..."}
						], 
			 "dc:creation": "YYY-MM-DD:HH:mm:ss", 
			 "dc:modif": "YYY-MM-DD:HH:mm:ss", 
			 "dc:ava": "YYY-MM-DD:HH:mm:ss", 
			 "dc:typ": [{"id": "value", "title": "texte"}
						], 
			 "dc:ext": "HH:MM:SS", 
			 "dc:loc": [{"lat": numvalue, "lng": numvalue}
						], 
			"dc:lic": "text", 
			"dc:rH": [{"id": "value", "title": "text"}
						], 
			"dc:aud": "text"
			}
	}, 
	...
]

```

How can I manage in the filter section (of logstash configuration file) to get an output like this :

```auto
   "metas": {"title": "text...", 
			 "id": "idvalue",
			 "subject": {{"id": "idvalue", "title": "text"}, 
							{"id": "idvalue", "title": "text"}, 
							{"id4": "idvalue", "title": "text..."}, 
							{"id": "idvalue", "title": "text"}
							}, 
			 "description": "long text.\r\n\r\n\r\n\r\n\r\n\r\n", 
			 "pub": {{"id": "idvalue", "title": "text..."}
						}, 
			 "creation": "YYY-MM-DD:HH:mm:ss", 
			 "modif": "YYY-MM-DD:HH:mm:ss", 
			 "ava": "YYY-MM-DD:HH:mm:ss", 
			 "typ": {{"id": "value", "title": "texte"}
						}, 
			 "ext": "HH:MM:SS", 
			 "loc": {{"lat": numvalue, "lng": numvalue}
						}, 
			"lic": "text", 
			"rH": {{"id": "value", "title": "text"}
						}, 
			"aud": "text"
			},
	   "metas": {"title": "text...", 
			 "id": "idvalue",
			 "subject": {{"id": "idvalue", "title": "text"}, 
							{"id": "idvalue", "title": "text"}, 
							{"id4": "idvalue", "title": "text..."}, 
							{"id": "idvalue", "title": "text"}
							}, 
			 "description": "long text.\r\n\r\n\r\n\r\n\r\n\r\n", 
			 "pub": {{"id": "idvalue", "title": "text..."}
						}, 
			 "creation": "YYY-MM-DD:HH:mm:ss", 
			 "modif": "YYY-MM-DD:HH:mm:ss", 
			 "ava": "YYY-MM-DD:HH:mm:ss", 
			 "typ": {{"id": "value", "title": "texte"}
						}, 
			 "ext": "HH:MM:SS", 
			 "loc": {{"lat": numvalue, "lng": numvalue}
						}, 
			"lic": "text", 
			"rH": {{"id": "value", "title": "text"}
						}, 
			"aud": "text"
			},
			....

```

Thanks!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 20, 2022, 6:01pm UTC](https://discuss.elastic.co/t/logstash-how-to-parse-and-split-nested-json-file/305308/2 "2022-05-20T18:01:29Z")

</div>

It sounds like you want a split filter to split the array into multiple events, and maybe a mutate+rename to move [metas] to the top-level, and mutate+remove\_field to get rid of the [header] field.

---

<div class="post-metadata">

**Author:** ![Iss](https://avatars.discourse-cdn.com/v4/letter/i/58956e/32.png) [@Iss](https://discuss.elastic.co/u/Iss)\
**Post date:** [May 20, 2022, 6:18pm UTC](https://discuss.elastic.co/t/logstash-how-to-parse-and-split-nested-json-file/305308/3 "2022-05-20T18:18:44Z")

</div>

Yes, that is what I want, a split filter as you say.  
Can you help with an example?  
Thank you!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 20, 2022, 7:05pm UTC](https://discuss.elastic.co/t/logstash-how-to-parse-and-split-nested-json-file/305308/4 "2022-05-20T19:05:18Z")

</div>

Is the JSON serialized in the [message] field or has it already been parsed? If so, what is the name of the field that contains the array?

---

<div class="post-metadata">

**Author:** ![Iss](https://avatars.discourse-cdn.com/v4/letter/i/58956e/32.png) [@Iss](https://discuss.elastic.co/u/Iss)\
**Post date:** [May 20, 2022, 7:52pm UTC](https://discuss.elastic.co/t/logstash-how-to-parse-and-split-nested-json-file/305308/5 "2022-05-20T19:52:58Z")

</div>

It is in the field [message]

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 20, 2022, 8:18pm UTC](https://discuss.elastic.co/t/logstash-how-to-parse-and-split-nested-json-file/305308/6 "2022-05-20T20:18:10Z")

</div>

OK, so if your message field looks like

```
   "message" => "[{\"header\": {\"id\": \"idvalue\", \n\t\t\t \"datestamp\": \"YYYY-MM-DD\"\n\t\t\t }, \n \"metas\": {\"dc:title\": \"text...\", ...]"

```

then you can use

```
    json { source => "message" target => "json" remove_field => ["message"] }
    split { field => "json" }
    mutate { rename => { "[json][metas]" => "metas" } }
    mutate { remove_field => ["json"] }
```

---

<div class="post-metadata">

**Author:** ![Iss](https://avatars.discourse-cdn.com/v4/letter/i/58956e/32.png) [@Iss](https://discuss.elastic.co/u/Iss)\
**Post date:** [May 21, 2022, 4:59pm UTC](https://discuss.elastic.co/t/logstash-how-to-parse-and-split-nested-json-file/305308/7 "2022-05-21T16:59:56Z")

</div>

> [@Badger](#):
>
> `"[{\"header\": {\"id\": \"idvalue\", \n\t\t\t \"datestamp\": \"YYYY-MM-DD\"\n\t\t\t }, \n \"metas\": {\"dc:title\": \"text...\", ...]"`

Sorry for the delay, I ran into a technical complication just after my last post.  
I tried with your code but it doesn't seem to work with my file.

I can see that "\n\t\t\t" or "\n" is missing before "metas". I don't have the following

```auto
"message" => "[{\"header\": {\"id\": \"idvalue\", \"datestamp\": \"YYYY-MM-DD\"\n\t\t\t }, \n \"metas\": {\"dc:title\": \"text...\", ...]"

```

but this one :

```auto
[{\"header\": {\"id\": \"idvalue\", \"datestamp\": \"YYYY-MM-DD" }, \"metas\": {\"dc:title\": \"text...\", ...]"

```

Is it the problem?

Thanks

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 21, 2022, 5:16pm UTC](https://discuss.elastic.co/t/logstash-how-to-parse-and-split-nested-json-file/305308/8 "2022-05-21T17:16:44Z")

</div>

> [@Iss](#):
>
> Is it the problem?

No, whitespace does not matter.

---

<div class="post-metadata">

**Author:** ![Iss](https://avatars.discourse-cdn.com/v4/letter/i/58956e/32.png) [@Iss](https://discuss.elastic.co/u/Iss)\
**Post date:** [May 23, 2022, 9:18am UTC](https://discuss.elastic.co/t/logstash-how-to-parse-and-split-nested-json-file/305308/9 "2022-05-23T09:18:01Z")

</div>

> [@Badger](#):
>
> `"message" => "[{\"header\": {\"id\`

I still haven't succeeded.  
After trying with your code I can see with debug and trace the following:

```auto
[WARN] 2022-05-23 11:20:54.407 [[main]>worker6] json - Error parsing json {:source=>"message", :raw=>"[{\"header\":...", :exception=>#<LogStash::Json::ParserError: Illegal unquoted character ((CTRL-CHAR, code 10)): has to be escaped using backslash to be included in string value
 at [Source: (byte[])"[{"header": ..."[truncated 138 bytes]; line: 1, column: 625]>}

[WARN] 2022-05-23 11:20:54.414 [[main]>worker6] split - Only String and Array types are splittable. field:json is of type = NilClass

```

```auto
{
          "tags" => [
        [0] "_jsonparsefailure",
        [1] "_split_type_failure"
    ],
"event" => {
"original" => "[{\"header\": {\"id\": \"idvalue\", \n\t\t\t \"datestamp\": \"YYYY-MM-DD\"\n\t\t\t }, \n \"metas\": {\"dc:title\": \"text...\", ... 
},
,
          "host" => {
        "name" => "D...."
    },
    "@timestamp" => 2022-...,
           "log" => {
        "file" => {
            "path" => "/simple.txt"
        }
    },

"message" => "[{\"header\": {\"id\": \"idvalue\", \n\t\t\t \"datestamp\": \"YYYY-MM-DD\"\n\t\t\t }, \n \"metas\": {\"dc:title\": \"text...\", ... ,
      "@version" => "1",
          "type" => "json"
}

```

Where is the problem?  
Thanks

---

<div class="post-metadata">

**Author:** ![Iss](https://avatars.discourse-cdn.com/v4/letter/i/58956e/32.png) [@Iss](https://discuss.elastic.co/u/Iss)\
**Post date:** [May 23, 2022, 10:52am UTC](https://discuss.elastic.co/t/logstash-how-to-parse-and-split-nested-json-file/305308/10 "2022-05-23T10:52:32Z")

</div>

If any question about my original data in the file please see link [Reproduction - Pastebin.com](https://pastebin.com/vmbKzNcr) for reproduction  
Thanks !

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 20, 2022, 10:53am UTC](https://discuss.elastic.co/t/logstash-how-to-parse-and-split-nested-json-file/305308/11 "2022-06-20T10:53:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.