# Parsing a pretty formatted JSON file using Logstash

**URL:** <https://discuss.elastic.co/t/parsing-a-pretty-formatted-json-file-using-logstash/224224>\
**Category:** Logstash\
**Created:** [March 19, 2020, 7:25am UTC](https://discuss.elastic.co/t/parsing-a-pretty-formatted-json-file-using-logstash/224224 "2020-03-19T07:25:07Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Noushad\_Aslam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/noushad_aslam/32/52004_2.png) [@Noushad\_Aslam](https://discuss.elastic.co/u/Noushad_Aslam)\
**Post date:** [March 19, 2020, 7:25am UTC](https://discuss.elastic.co/t/parsing-a-pretty-formatted-json-file-using-logstash/224224/1 "2020-03-19T07:25:08Z")

</div>

Hi,

I am pretty new to Elastic stack. I have a need to index a JSON file data. Below is my JSON file format.  
[ {  
"metricId" : 1776736,  
"metricName" : "Hardware Resources|Memory|Used %",  
"metricPath" : "Application Infrastructure Performance|Root|Individual Nodes|FBLQAMT|Hardware Resources|Memory|Used %",  
"frequency" : "TEN\_MIN",  
"metricValues" : [ {  
"startTimeInMillis" : 1582555800000,  
"occurrences" : 1,  
"current" : 25,  
"min" : 25,  
"max" : 25,  
"useRange" : true,  
"count" : 20,  
"sum" : 500,  
"value" : 25,  
"standardDeviation" : 0  
}, {  
"startTimeInMillis" : 1582556400000,  
....  
}]  
}, {  
"metricId" : 1776736,  
"metricName" : "Hardware Resources|Memory|Used %",  
"metricPath" : "Application Infrastructure Performance|Root|Individual Nodes|FBLTESTUXP|Hardware Resources|Memory|Used %",  
"frequency" : "TEN\_MIN",  
"metricValues" : [ {  
"startTimeInMillis" : 1582555800000,  
.....  
} ]  
} ]

Can anyone please provide me a logstash configuration file for the above JSON file?

---

<div class="post-metadata">

**Author:** ![Dhruvil\_Doshi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dhruvil_doshi/32/50639_2.png) [@Dhruvil\_Doshi](https://discuss.elastic.co/u/Dhruvil_Doshi)\
**Post date:** [March 19, 2020, 7:41am UTC](https://discuss.elastic.co/t/parsing-a-pretty-formatted-json-file-using-logstash/224224/2 "2020-03-19T07:41:27Z")

</div>

> [@Noushad\_Aslam](#):
>
> ,

Try this

```auto
input { 
	file {

		start_position => "beginning"		
		path => "pathOfYourJsonFile"	
		sincedb_path => "/dev/null"
		codec => "json"
	}
}

output {
	elasticsearch {						
		hosts => ["Ip_Address_of_Elasticsearch:Port_No"] # localhost:9200
		index =>"Index_Name"
  	}
}

```

---

<div class="post-metadata">

**Author:** ![Noushad\_Aslam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/noushad_aslam/32/52004_2.png) [@Noushad\_Aslam](https://discuss.elastic.co/u/Noushad_Aslam)\
**Post date:** [March 19, 2020, 8:28am UTC](https://discuss.elastic.co/t/parsing-a-pretty-formatted-json-file-using-logstash/224224/3 "2020-03-19T08:28:37Z")

</div>

Thank you for the reply. I tried the below configuration.

```
input {  
file 
{
    path => "H:/ELK/AppD_Data/FBLQAFeb24_50uMem.json"
    start_position => "beginning"
	sincedb_path => "nul"
	codec => json
  }
}

filter 
{

}

output {
	stdout { codec => rubydebug }
#	elasticsearch {
# hosts => ["localhost:9200"]
# index => "appdynamics_metrics2"
#		
#	}
#	
}

```

I am getting the below error.

```
[2020-03-19T13:54:12,274][ERROR][logstash.codecs.json] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (String)"[{"; line: 1, column: 3])
 at [Source: (String)"[{"; line: 1, column: 7]>, :data=>"[ {"}

```

I need each nested fields to be fields in the output. Any suggestions?

---

<div class="post-metadata">

**Author:** ![Dhruvil\_Doshi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dhruvil_doshi/32/50639_2.png) [@Dhruvil\_Doshi](https://discuss.elastic.co/u/Dhruvil_Doshi)\
**Post date:** [March 19, 2020, 9:43am UTC](https://discuss.elastic.co/t/parsing-a-pretty-formatted-json-file-using-logstash/224224/4 "2020-03-19T09:43:00Z")

</div>

Hi Can you show me the message field ? If possible also show the contents of the file. In file, every field of JSON object is written in new line like you have showed it above

---

<div class="post-metadata">

**Author:** ![Noushad\_Aslam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/noushad_aslam/32/52004_2.png) [@Noushad\_Aslam](https://discuss.elastic.co/u/Noushad_Aslam)\
**Post date:** [March 19, 2020, 12:11pm UTC](https://discuss.elastic.co/t/parsing-a-pretty-formatted-json-file-using-logstash/224224/5 "2020-03-19T12:11:52Z")

</div>

Thanks for the help. It is wring the error as below after logstash is started

```
[2020-03-19T17:31:23,482][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600}
[2020-03-19T17:31:23,618][ERROR][logstash.codecs.json] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (String)"[{"; line: 1, column: 3])
 at [Source: (String)"[{"; line: 1, column: 7]>, :data=>"[ {"}
[2020-03-19T17:31:23,666][ERROR][logstash.codecs.json] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: incompatible json object type=java.lang.String , only hash map or arrays are supported>, :data=>" \"metricId\" : 1776736,"}
[2020-03-19T17:31:23,669][ERROR][logstash.codecs.json] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: incompatible json object type=java.lang.String , only hash map or arrays are supported>, :data=>" \"metricName\" : \"Hardware Resources|Memory|Used %\","}
[2020-03-19T17:31:23,671][ERROR][logstash.codecs.json] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: incompatible json object type=java.lang.String , only hash map or arrays are supported>, :data=>" \"metricPath\" : \"Application Infrastructure Performance|Root|Individual Nodes|FBLQAMT|Hardware Resources|Memory|Used %\","}
[2020-03-19T17:31:23,673][ERROR][logstash.codecs.json] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: incompatible json object type=java.lang.String , only hash map or arrays are supported>, :data=>" \"frequency\" : \"TEN_MIN\","}

```

After many errors i see the below

```
[2020-03-19T17:31:26,032][ERROR][logstash.codecs.json] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: incompatible json object type=java.lang.String , only hash map or arrays are supported>, :data=>" \"useRange\" : true,"}
{
          "path" => "H:/ELK/AppD_Data/FBLQAFeb24_50uMem.json",
          "host" => "PVDI-FCBT081",
          "tags" => [
        [0] "_jsonparsefailure"
    ],
       "message" => "[ {",
    "@timestamp" => 2020-03-19T12:01:23.634Z,
      "@version" => "1"
}
{
          "path" => "H:/ELK/AppD_Data/FBLQAFeb24_50uMem.json",
          "host" => "PVDI-FCBT081",
          "tags" => [
        [0] "_jsonparsefailure"
    ],
       "message" => " \"metricPath\" : \"Application Infrastructure Performance|Root|Individual Nodes|FBLQAMT|Hardware Resources|Memory|Used %\",",
    "@timestamp" => 2020-03-19T12:01:23.672Z,
      "@version" => "1"
}

```

It is writing each line in the message field. I need each field in the data to be seperate  
in Kibana for analysis.  
I have mentioned the input JSON file at the top. which looks like below.

```
[ {
"metricId" : 1776736,
"metricName" : "Hardware Resources|Memory|Used %",
"metricPath" : "Application Infrastructure Performance|Root|Individual Nodes|FBLQAMT|Hardware Resources|Memory|Used %",
"frequency" : "TEN_MIN",
"metricValues" : [ {
"startTimeInMillis" : 1582555800000,
"occurrences" : 1,
"current" : 25,
"min" : 25,
"max" : 25,
"useRange" : true,
"count" : 20,
"sum" : 500,
"value" : 25,
"standardDeviation" : 0
}, {
"startTimeInMillis" : 1582556400000,
....
}]
}, {
"metricId" : 1776736,
"metricName" : "Hardware Resources|Memory|Used %",
"metricPath" : "Application Infrastructure Performance|Root|Individual Nodes|FBLTESTUXP|Hardware Resources|Memory|Used %",
"frequency" : "TEN_MIN",
"metricValues" : [ {
"startTimeInMillis" : 1582555800000,
.....
} ]
} ]
```

---

<div class="post-metadata">

**Author:** ![jbpratt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jbpratt/32/57424_2.png) [@jbpratt](https://discuss.elastic.co/u/jbpratt)\
**Post date:** [March 19, 2020, 12:25pm UTC](https://discuss.elastic.co/t/parsing-a-pretty-formatted-json-file-using-logstash/224224/6 "2020-03-19T12:25:25Z")

</div>

Hi, @Noushad_Aslam

I wonder if attempting something like [this](https://discuss.elastic.co/t/remove-n-and-t-from-message/142620) would work.

The "multiline" plugin may also be able to solve your problem:  
[https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html)

Best of luck!

---

<div class="post-metadata">

**Author:** ![Noushad\_Aslam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/noushad_aslam/32/52004_2.png) [@Noushad\_Aslam](https://discuss.elastic.co/u/Noushad_Aslam)\
**Post date:** [March 19, 2020, 1:08pm UTC](https://discuss.elastic.co/t/parsing-a-pretty-formatted-json-file-using-logstash/224224/7 "2020-03-19T13:08:09Z")

</div>

@jbpratt I tried the gsub but still each line is written in the message field.  
removed json codec and used json filter, still same problem.  
Can you please provide a correct logstash configuration file for the above mentioned JSON file input?  
The field `"metricValues"` is an array of nested fields like `"startTimeInMillis"`, `"occurrences"` etc..

Thanks in advance.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 22, 2020, 4:24pm UTC](https://discuss.elastic.co/t/parsing-a-pretty-formatted-json-file-using-logstash/224224/8 "2020-03-22T16:24:45Z")

</div>

If you want to consume the entire file as a single event you must use a multiline codec. See [this](https://discuss.elastic.co/t/parsing-array-of-json-objects-with-logstash-and-injesting-to-elastic/203197/2) post for an example.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 19, 2020, 4:24pm UTC](https://discuss.elastic.co/t/parsing-a-pretty-formatted-json-file-using-logstash/224224/9 "2020-04-19T16:24:46Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.