# Nested Json logs - how to filter correctly **URL:** \ **Category:** Logstash\ **Created:** [March 23, 2016, 2:37pm UTC](https://discuss.elastic.co/t/nested-json-logs-how-to-filter-correctly/45234 "2016-03-23T14:37:04Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![yuv14](https://avatars.discourse-cdn.com/v4/letter/y/e5b9ba/32.png) [@yuv14](https://discuss.elastic.co/u/yuv14)\ **Post date:** [March 23, 2016, 2:37pm UTC](https://discuss.elastic.co/t/nested-json-logs-how-to-filter-correctly/45234/1 "2016-03-23T14:37:04Z")
Hi, I'm trying to input a complex Json log with nested fields. Here is my .conf file: `input { file { path => /home/logstash-user/json-try.json" start_position => "beginning" codec => "json" } } output { elasticsearch { codec => "json"} stdout { codec => "rubydebug"} }` I tried to input to files, one simple with 1 nested object, and another one more complex with about 3 nested objects. The first file is filtered correctly, but the second one isn't filtered at all, and the following error appears: "Attempted to send a bulk request to Elasticsearch .... but an error occurred and it failed!" and so on... What is my problem? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 23, 2016, 8:09pm UTC](https://discuss.elastic.co/t/nested-json-logs-how-to-filter-correctly/45234/2 "2016-03-23T20:09:05Z")
The elasticsearch output should not have JSON codec as this is handled internally. Remove this and try again. ---
**Author:** ![yuv14](https://avatars.discourse-cdn.com/v4/letter/y/e5b9ba/32.png) [@yuv14](https://discuss.elastic.co/u/yuv14)\ **Post date:** [March 24, 2016, 7:26am UTC](https://discuss.elastic.co/t/nested-json-logs-how-to-filter-correctly/45234/3 "2016-03-24T07:26:00Z")
Hi, thank you for your replay. I removed the JSON codec from the elasticsearch output, but nothing changed. Here is the input that worked: `{"one":{"id":1, "name":"john", "job":"programmer"}, "two":{"id":2, "name":"david", "job":"programmer"}}` Here is the complex input that didn't work: `{"project":{"metadata":{"project_name":"calc", "date":"01.01.16"}, "details":{"feature 1":{"name":"gui", "description":"design"}, "feature 2":"modulo", "description":"modulo"}}}}` What might be the problem? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 24, 2016, 7:34am UTC](https://discuss.elastic.co/t/nested-json-logs-how-to-filter-correctly/45234/4 "2016-03-24T07:34:06Z")
The last entry seems to have a curly brace too many at the end and is therefore nat valid JSON. ---
**Author:** ![yuv14](https://avatars.discourse-cdn.com/v4/letter/y/e5b9ba/32.png) [@yuv14](https://discuss.elastic.co/u/yuv14)\ **Post date:** [March 24, 2016, 8:45am UTC](https://discuss.elastic.co/t/nested-json-logs-how-to-filter-correctly/45234/5 "2016-03-24T08:45:02Z")
thanks, I fixed it. But now it doesn't parse the json, it just puts the whole input into the "message" field, instead of parsing it by the json objects. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 24, 2016, 9:27am UTC](https://discuss.elastic.co/t/nested-json-logs-how-to-filter-correctly/45234/6 "2016-03-24T09:27:41Z")
Do you still have got the JSON codec for the file input? Can you share you current config? ---
**Author:** ![yuv14](https://avatars.discourse-cdn.com/v4/letter/y/e5b9ba/32.png) [@yuv14](https://discuss.elastic.co/u/yuv14)\ **Post date:** [March 24, 2016, 9:48am UTC](https://discuss.elastic.co/t/nested-json-logs-how-to-filter-correctly/45234/7 "2016-03-24T09:48:35Z")
`input { file { path => /home/logstash-user/json-try.json" start_position => "beginning" codec => "json" } } output { elasticsearch { } stdout { codec => "rubydebug"} }` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 24, 2016, 9:49am UTC](https://discuss.elastic.co/t/nested-json-logs-how-to-filter-correctly/45234/8 "2016-03-24T09:49:28Z")
What does the ruby debug output look like for the event? ---
**Author:** ![yuv14](https://avatars.discourse-cdn.com/v4/letter/y/e5b9ba/32.png) [@yuv14](https://discuss.elastic.co/u/yuv14)\ **Post date:** [March 24, 2016, 11:04am UTC](https://discuss.elastic.co/t/nested-json-logs-how-to-filter-correctly/45234/9 "2016-03-24T11:04:56Z")
Logstash startup completed { "message" =\> "{"project":{"metadata":{"project\_name":"calc", "date":"01.01.16"}, "details":{"feature 1":{"name":"gui", "description":"design"}, "feature 2":"modulo", "description":"modulo"}}}", "tags" =\> [ [0] "\_jsonparsefailure" ], "@version" =\> "1", "timestamp" =\> "2016-03-24T08:32:50.667Z", "host" =\> "elk" "path" =\> "/home/logstash-user/json-try.json" } ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 24, 2016, 11:44am UTC](https://discuss.elastic.co/t/nested-json-logs-how-to-filter-correctly/45234/10 "2016-03-24T11:44:58Z")
That can not be parsed by the codec as it is not valid JSON (indicated by the `_jsonparsefailure` tag). If you can not correct the generation at the source, you may need to load it using a plain codec and then transform it into valid JSON before applying a JSON filter instead. ---
**Author:** ![yuv14](https://avatars.discourse-cdn.com/v4/letter/y/e5b9ba/32.png) [@yuv14](https://discuss.elastic.co/u/yuv14)\ **Post date:** [March 24, 2016, 12:10pm UTC](https://discuss.elastic.co/t/nested-json-logs-how-to-filter-correctly/45234/11 "2016-03-24T12:10:41Z")
I put the json into a json validator, had a minor fix in it. Now I get a huge error in red and yellow, the same one I wrote about in the beginning. It looks something like this: "Attempted to send bulk request to Elasticsearch configured at (configuration details...) but an error occurred and it failed!" and so on, it's pretty long... The same error is printed about 4 times. ---
**Author:** ![yuv14](https://avatars.discourse-cdn.com/v4/letter/y/e5b9ba/32.png) [@yuv14](https://discuss.elastic.co/u/yuv14)\ **Post date:** [March 24, 2016, 12:23pm UTC](https://discuss.elastic.co/t/nested-json-logs-how-to-filter-correctly/45234/12 "2016-03-24T12:23:23Z")
Hi Christian, In my searches around the web I found someone saying they had a problem to filter input with dots (".") in it, and in one of the inputs I indeed had a dot. I removed it and everything worked fine. Do you know the cause of this problem? Or how to overcome it? Thanks ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 5:05am UTC](https://discuss.elastic.co/t/nested-json-logs-how-to-filter-correctly/45234/13 "2017-07-06T05:05:34Z")