# Having problem with parsing Json data using Filebeat and Logstash

**URL:** <https://discuss.elastic.co/t/having-problem-with-parsing-json-data-using-filebeat-and-logstash/101683>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [September 25, 2017, 10:52am UTC](https://discuss.elastic.co/t/having-problem-with-parsing-json-data-using-filebeat-and-logstash/101683 "2017-09-25T10:52:10Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Shubham\_Mahajan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shubham_mahajan/32/22140_2.png) [@Shubham\_Mahajan](https://discuss.elastic.co/u/Shubham_Mahajan)\
**Post date:** [September 25, 2017, 10:52am UTC](https://discuss.elastic.co/t/having-problem-with-parsing-json-data-using-filebeat-and-logstash/101683/1 "2017-09-25T10:52:10Z")

</div>

i have this type of json logs:

`1485907213693 {"counter_name": "image", "resource_id": "999d4b25-0539-419e-9991-42f2cf26061b", "timestamp": "2017-02-01T00:00:40Z", "counter_volume": 1, "user_id": null, "message_signature": "f9cc0305c9fc95ba9761aded3df342af4e03e8fccc898b1acfd20d6a6041d49d", "resource_metadata": {"status": "active", "name": "OfficialD20.7_v7_raw", "deleted": false, "container_format": "bare", "created_at": "2016-11-11T21:03:21.000000", "disk_format": "raw", "updated_at": "2016-11-15T22:45:51.000000", "protected": false, "min_ram": 0, "checksum": "a185ae0b7314e53d5256", "min_disk": 0, "is_public": false, "deleted_at": null, "properties": {"description": null}, "size": 2147483648}, "source": "open", "counter_unit": "image", "project_id": "03f13ee524ee4ea2afabc839014ff7f2", "message_id": "78209f86-e811-11e6-b78f-0cc47a6431ab", "counter_type": "gauge"}`

If I try toparse this using json codec, it's giving me this error:

`JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: incompatible json object type=java.lang.Long , only hash map or arrays are supported>,`

I know that i have a long number before every json object, any idea how to parse this???

Thanks in advance!!!

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 25, 2017, 11:01am UTC](https://discuss.elastic.co/t/having-problem-with-parsing-json-data-using-filebeat-and-logstash/101683/2 "2017-09-25T11:01:45Z")

</div>

You will need to first separate out the initial timestamp from the rest, e.g. using a grok filter, before you can apply a json filter to the rest. The json codec will not work with data in this format as it is not all valid JSON.

---

<div class="post-metadata">

**Author:** ![Shubham\_Mahajan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shubham_mahajan/32/22140_2.png) [@Shubham\_Mahajan](https://discuss.elastic.co/u/Shubham_Mahajan)\
**Post date:** [September 25, 2017, 12:46pm UTC](https://discuss.elastic.co/t/having-problem-with-parsing-json-data-using-filebeat-and-logstash/101683/3 "2017-09-25T12:46:13Z")

</div>

> [@Shubham\_Mahajan](#):
>
> 1485907213693

what timestamp are u talking about???  
**1485907213693** This no. is the only thing i need to separate right??

which filter to use for that...??

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 25, 2017, 12:58pm UTC](https://discuss.elastic.co/t/having-problem-with-parsing-json-data-using-filebeat-and-logstash/101683/4 "2017-09-25T12:58:58Z")

</div>

That looks like a millisecond timestamp. Use a [dissect filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-dissect.html) or a [grok filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html) to store the timestamp in one field and the rest in another that you can then apply a json filter on.

---

<div class="post-metadata">

**Author:** ![Shubham\_Mahajan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shubham_mahajan/32/22140_2.png) [@Shubham\_Mahajan](https://discuss.elastic.co/u/Shubham_Mahajan)\
**Post date:** [September 25, 2017, 1:06pm UTC](https://discuss.elastic.co/t/having-problem-with-parsing-json-data-using-filebeat-and-logstash/101683/5 "2017-09-25T13:06:20Z")

</div>

yeah thanks Christian. I'll try that.

---

<div class="post-metadata">

**Author:** ![Shubham\_Mahajan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shubham_mahajan/32/22140_2.png) [@Shubham\_Mahajan](https://discuss.elastic.co/u/Shubham_Mahajan)\
**Post date:** [September 25, 2017, 1:38pm UTC](https://discuss.elastic.co/t/having-problem-with-parsing-json-data-using-filebeat-and-logstash/101683/6 "2017-09-25T13:38:26Z")

</div>

hey Christian, I tried to parse these logs **(removed the timestamp that was before json)**

`{"counter_name": "image", "resource_id": "999d4b25-0539-419e-9991-42f2cf26061b", "timestamp": "2017-02-01T00:00:40Z", "counter_volume": 1, "user_id": null, "message_signature": "f9cc0305c9fc95ba9761aded3df342af4e03e8fccc898b1acfd20d6a6041d49d", "resource_metadata": {"status": "active", "name": "OfficialD20.7_v7_raw", "deleted": false, "container_format": "bare", "created_at": "2016-11-11T21:03:21.000000", "disk_format": "raw", "updated_at": "2016-11-15T22:45:51.000000", "protected": false, "min_ram": 0, "checksum": "a185ae0b7314e53d5256", "min_disk": 0, "is_public": false, "deleted_at": null, "properties": {"description": null}, "size": 2147483648}, "source": "open", "counter_unit": "image", "project_id": "03f13ee524ee4ea2afabc839014ff7f2", "message_id": "78209f86-e811-11e6-b78f-0cc47a6431ab", "counter_type": "gauge"}`

Here filebeat sent the events to logstash, but logstash printed this in the console and didn't send any data to ES or create any Index.

**Message on Logstash console:**

```
2017-09-25T13:29:40.883Z L9608 %{message}
2017-09-25T13:29:40.883Z L9608 %{message}
2017-09-25T13:29:40.883Z L9608 %{message}
2017-09-25T13:29:40.883Z L9608 %{message}
2017-09-25T13:29:40.883Z L9608 %{message}
2017-09-25T13:29:40.883Z L9608 %{message}
2017-09-25T13:29:40.883Z L9608 %{message}
2017-09-25T13:29:40.883Z L9608 %{message}
2017-09-25T13:29:40.883Z L9608 %{message}
2017-09-25T13:29:40.883Z L9608 %{message}
2017-09-25T13:29:40.883Z L9608 %{message}

```

**Logstash.conf**

```
input {
	beats {
		port => 5044
		codec => json
		type => ceilometer
	}
}

filter {
	if [type] == "ceilometer" {
		date {
			match => ["timestamp", "YYYY-MM-dd'T'HH:mm:ss'Z'"]
			remove_field => "timestamp"
			timezone => "UTC"
		}
		date {
			match => ["[resource_metadata][created_at]", "YYYY-MM-dd HH:mm:ss.SSSSSS"]
			remove_field => "[resource_metadata][created_at]"
			target => "[resource_metadata][created_at_parsed]"
			timezone => "UTC"
		}
		
	}
}

output {
	stdout { }
	elasticsearch {
		hosts => ["http://localhost:9200"]
		index => "Ceilometer_logs"
		document_type => "json_ceilometer_type"
	}
}

```

**Filebeat.yml**

```
filebeat.prospectors:

- input_type: log

  paths:
    #- /var/log/*.log
    - \Users\ab66415\Desktop\ELK\dileep_docs\ceilometer_logs.txt

scan_frequency: 5s

backoff: 3s

json.keys_under_root: true
json.add_error_key: true
close_inactive: 5m

output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

```

Any idea why it's happening???

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 25, 2017, 1:44pm UTC](https://discuss.elastic.co/t/having-problem-with-parsing-json-data-using-filebeat-and-logstash/101683/7 "2017-09-25T13:44:31Z")

</div>

While troubleshooting the Logstash configuration, comment out the elasticsearch output and only use a stdout output with a `rubydebug` codec. This will show you the structure of the events you are processing and make it easier to find issues.

---

<div class="post-metadata">

**Author:** ![Shubham\_Mahajan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shubham_mahajan/32/22140_2.png) [@Shubham\_Mahajan](https://discuss.elastic.co/u/Shubham_Mahajan)\
**Post date:** [September 25, 2017, 1:52pm UTC](https://discuss.elastic.co/t/having-problem-with-parsing-json-data-using-filebeat-and-logstash/101683/8 "2017-09-25T13:52:38Z")

</div>

> [@Christian\_Dahlqvist](#):
>
> rubydebug

i am getting these two errors, Could you suggest something

```
failed to parse [resource_metadata.created_at]

Invalid format: "2016-11-21 22:51:59+00:00" is malformed at " 22:51:59+00:00"

```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 25, 2017, 1:56pm UTC](https://discuss.elastic.co/t/having-problem-with-parsing-json-data-using-filebeat-and-logstash/101683/9 "2017-09-25T13:56:21Z")

</div>

Comment out your date filters so you can see the structure of the event without any errors. you may also need to comment out the json codec in the beats input if that turns out to be causing problems. Then add filter by filter until it works as you want it to.

---

<div class="post-metadata">

**Author:** ![Shubham\_Mahajan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shubham_mahajan/32/22140_2.png) [@Shubham\_Mahajan](https://discuss.elastic.co/u/Shubham_Mahajan)\
**Post date:** [September 25, 2017, 1:56pm UTC](https://discuss.elastic.co/t/having-problem-with-parsing-json-data-using-filebeat-and-logstash/101683/10 "2017-09-25T13:56:59Z")

</div>

Thanks a lot, christian!! 🙂

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 25, 2017, 1:58pm UTC](https://discuss.elastic.co/t/having-problem-with-parsing-json-data-using-filebeat-and-logstash/101683/11 "2017-09-25T13:58:22Z")

</div>

If you encounter further issues and need assistance, it helps if you can show what the resulting event looks like.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 23, 2017, 1:59pm UTC](https://discuss.elastic.co/t/having-problem-with-parsing-json-data-using-filebeat-and-logstash/101683/12 "2017-10-23T13:59:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.