# Parse json data with logstash

**URL:** <https://discuss.elastic.co/t/parse-json-data-with-logstash/297792>\
**Category:** Logstash\
**Created:** [February 21, 2022, 4:52pm UTC](https://discuss.elastic.co/t/parse-json-data-with-logstash/297792 "2022-02-21T16:52:06Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![sylar11](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylar11/32/102086_2.png) [@sylar11](https://discuss.elastic.co/u/sylar11)\
**Post date:** [February 21, 2022, 4:52pm UTC](https://discuss.elastic.co/t/parse-json-data-with-logstash/297792/1 "2022-02-21T16:52:06Z")

</div>

Hi all,  
I'm a newbie to logstash and I'm trying to parse a JSON data file like the following

```auto
{
    "A001": {
        "X": 503744.7,
        "Y": 4726339.0,
        "Z": 458.84,
        "LON": -2.954286956913572,
        "LAT": 42.68952475979137,
        "dates": [
            "2015-01-01",
            "2015-01-02",
            "2015-01-03",
            "2015-01-04",
            "2015-01-05",
            "2015-01-06"            
        ],
        "values": [
            "56.9",
            "49.7",
            "48.1",
            "37.1",
            "34.4",
            "35.9"         
        ]
    },
    "A002": {
        "X": 607870.5,
        "Y": 4670754.0,
        "Z": 264.83,
        "LON": -1.69378623727067,
        "LAT": 42.18149989583031,
        "dates": [
            "2015-01-01",
            "2015-01-02",
            "2015-01-03",
            "2015-01-04"          
        ],
        "values": [
            "287",
            "231",
            "207",
            "191"
		]	
    },
    "A403": {
        "X": 868708.0,
        "Y": 4709148.0,
        "Z": 849.0,
        "LON": 1.483146867002623,
        "LAT": 42.44694604132231,
        "dates": [
            "2015-01-01",
            "2015-01-02",
            "2015-01-03",
            "2015-01-04",
            "2015-01-05",
            "2015-01-06",
            "2015-01-07",
            "2015-01-08",
            "2015-01-09"            
        ],
        "values": [
            "2.296",
            "7.033",
            "2.298",
            "2.275",
            "7.207",
            "5.456",
            "4.794",
            "4.24",
            "4.748"
        ]
    }
}

```

and I'm using the following .conf file

```auto
input {
        file {
                type => "json"
                start_position => "beginning"
                path => "/etc/logstash/json-data.json"
                sincedb_path => "/dev/null" #to force to parse from the beginning
                codec => "json_lines"
        }
}

filter {
        json {
                source => "message"
        }
}

output {

        elasticsearch {
                hosts => ["https://ip:9200"]
                user => "elastic"
                password => "mypassword"
                ssl_certificate_verification => false
        index => "demo-json"
        document_type => "json"
        }

        stdout { codec => rubydebug }

```

but I'm getting this error:

```auto
 JSON parse error, original data now in message field {:message=>"incompatible json object type=java.lang.String , only hash map or arrays are supported", 

```

Could you please tell me what I'm doing wrong?

thanks in advance

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 21, 2022, 5:20pm UTC](https://discuss.elastic.co/t/parse-json-data-with-logstash/297792/2 "2022-02-21T17:20:37Z")

</div>

What is the full error message? Why are you using a json filter if you have already parsed the JSON using a json\_lines codec?

---

<div class="post-metadata">

**Author:** ![sylar11](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylar11/32/102086_2.png) [@sylar11](https://discuss.elastic.co/u/sylar11)\
**Post date:** [February 22, 2022, 7:25am UTC](https://discuss.elastic.co/t/parse-json-data-with-logstash/297792/3 "2022-02-22T07:25:43Z")

</div>

Hi @Badger  
this is the error I got in /var/log/logstash/logstash-plain.log

```auto
[WARN][logstash.filters.json][main][798c76c6b7564c05f1f7e4bd3babb64bebdec2ed56d7e2d2a44c15d6f4372356] Error parsing json {:source=>"message", :raw=>" \"Z\": 849.0,", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (byte[])" "Z": 849.0,"; line: 1, column: 13]>}
[2022-02-21T16:45:57,504][WARN][logstash.filters.json][main][798c76c6b7564c05f1f7e4bd3babb64bebdec2ed56d7e2d2a44c15d6f4372356] Error parsing json {:source=>"message", :raw=>" \"LON\": 1.483146867002623,", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (byte[])" "LON": 1.483146867002623,"; line: 1, column: 15]>}
[2022-02-21T16:45:57,504][WARN][logstash.filters.json][main][798c76c6b7564c05f1f7e4bd3babb64bebdec2ed56d7e2d2a44c15d6f4372356] Error parsing json {:source=>"message", :raw=>" }", :exception=>#<LogStash::Json::ParserError: Unexpected close marker '}': expected ']' (for root starting at [Source: (byte[])" }"; line: 1, column: 0])
 at [Source: (byte[])" }"; line: 1, column: 6]>}
[2022-02-21T16:45:57,505][WARN][logstash.filters.json][main][798c76c6b7564c05f1f7e4bd3babb64bebdec2ed56d7e2d2a44c15d6f4372356] Error parsing json {:source=>"message", :raw=>" \"LAT\": 42.44694604132231,", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (byte[])" "LAT": 42.44694604132231,"; line: 1, column: 15]>}
[2022-02-21T16:45:57,505][WARN][logstash.filters.json][main][798c76c6b7564c05f1f7e4bd3babb64bebdec2ed56d7e2d2a44c15d6f4372356] Error parsing json {:source=>"message", :raw=>" \"4.24\",", :exception=>#<LogStash::Json::ParserError: Unexpected character (',' (code 44)): expected a value
 at [Source: (byte[])" "4.24","; line: 1, column: 20]>}

```

About the json filter probably I misunderstood the documentation.

How should I arrange the conf file?

thanks

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 22, 2022, 6:18pm UTC](https://discuss.elastic.co/t/parse-json-data-with-logstash/297792/4 "2022-02-22T18:18:39Z")

</div>

See [this](https://discuss.elastic.co/t/parsing-json-file-with-logstash/214080/2) thread.

---

<div class="post-metadata">

**Author:** ![sylar11](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylar11/32/102086_2.png) [@sylar11](https://discuss.elastic.co/u/sylar11)\
**Post date:** [February 23, 2022, 9:27am UTC](https://discuss.elastic.co/t/parse-json-data-with-logstash/297792/5 "2022-02-23T09:27:08Z")

</div>

Dear @Badger  
following our suggestion now I'm able to correctly parse the JSON file. May I ask you why it is messing up a bit the order of the fields with respect to the input file? For example, the field LON is at the bottom or the field Z in the middle.

```auto
"@timestamp" => 2022-02-23T09:23:05.369615Z,
          "A403" => {
             "X" => 868708.0,
             "Y" => 4709148.0,
           "LAT" => 42.44694604132231,
        "values" => [
            [0] "2.296",
            [1] "7.033",
            [2] "2.298",
            [3] "2.275",
            [4] "7.207",
            [5] "5.456",
            [6] "4.794",
            [7] "4.24",
            [8] "4.748"
        ],
             "Z" => 849.0,
         "dates" => [
            [0] "2015-01-01",
            [1] "2015-01-02",
            [2] "2015-01-03",
            [3] "2015-01-04",
            [4] "2015-01-05",
            [5] "2015-01-06",
            [6] "2015-01-07",
            [7] "2015-01-08",
            [8] "2015-01-09"
        ],
           "LON" => 1.483146867002623
    },

```

thanks

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 23, 2022, 2:43pm UTC](https://discuss.elastic.co/t/parse-json-data-with-logstash/297792/6 "2022-02-23T14:43:22Z")

</div>

Why do you care about the order of the fields on the event?

---

<div class="post-metadata">

**Author:** ![sylar11](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylar11/32/102086_2.png) [@sylar11](https://discuss.elastic.co/u/sylar11)\
**Post date:** [February 23, 2022, 2:51pm UTC](https://discuss.elastic.co/t/parse-json-data-with-logstash/297792/7 "2022-02-23T14:51:54Z")

</div>

@Badger actually I was worried that something was still wrong that's why I was asking 😅 . If the order does not matter for possible next steps, then do not consider my question

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 23, 2022, 2:54pm UTC](https://discuss.elastic.co/t/parse-json-data-with-logstash/297792/8 "2022-02-23T14:54:12Z")

</div>

I do not believe that order matters.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 23, 2022, 2:54pm UTC](https://discuss.elastic.co/t/parse-json-data-with-logstash/297792/9 "2022-03-23T14:54:38Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.