# Parsing a long json line with filebeat to logstash **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [August 20, 2018, 7:54pm UTC](https://discuss.elastic.co/t/parsing-a-long-json-line-with-filebeat-to-logstash/145231 "2018-08-20T19:54:17Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Gab\_Ubi](https://avatars.discourse-cdn.com/v4/letter/g/b77776/32.png) [@Gab\_Ubi](https://discuss.elastic.co/u/Gab_Ubi)\ **Post date:** [August 20, 2018, 7:54pm UTC](https://discuss.elastic.co/t/parsing-a-long-json-line-with-filebeat-to-logstash/145231/1 "2018-08-20T19:54:18Z")
Hi all, Im trying to parse a big json line into logstash. The json is an array of multpiple object. I'd like each object to be an event and each value to be a field. This is my filebeat config: ``` filebeat.inputs: - type: log enabled: true # Paths that should be crawled and fetched. Glob based paths. paths: - "D:/data/*" processors: - decode_json_fields: fields: ["studio_state"] output.logstash: # The Logstash hosts hosts: ["logstash:5151"] ``` JSON string looks like (but is much more lenghty) ``` [{ "Location": "0.0,0.0", "event_name": "Resource Tab Selected", "main_category": "Sequencer", "sub_category": "Not available", "start_timestamp": "2018-08-19T00:17:04.847", "end_timestamp": "2018-08-19T00:17:04.847" }, { "Location": "0.0,0.0", "event_name": "Resource Tab Selected", "main_category": "Sequencer", "sub_category": "Not available", "start_timestamp": "2018-08-19T00:17:04.847", "end_timestamp": "2018-08-19T00:17:04.847" }] ``` And my logstash config: ``` input { tcp { port => 5000 } beats { codec => "json" port => 5151 } } filter { json { source => "message" } } ## Add your filters / logstash plugins configuration here output { elasticsearch { hosts => "elasticsearch:9200" index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" } } ``` I've tried many different combinations but with this one I get this message: `{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":343,"time":{"ms":78}},"total":{"ticks":389,"time":{"ms":78},"value":389},"user":{"ticks":46}},"info":{"ephemeral_id":"714c1baa-3aa1-4d39-8884-273d6edea87b","uptime":{"ms":180078}},"memstats":{"gc_next":4194304,"memory_alloc":1702152,"memory_total":3961352,"rss":12288}},"filebeat":{"events":{"added":1,"done":1},"harvester":{"open_files":3,"running":3,"started":1}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0,"filtered":1,"total":1}}},"registrar":{"states":{"current":3,"update":1},"writes":{"success":1,"total":1}}}}}` Most annoying part to me is : "events":{"active":0,"filtered":1,"total":1 I don't understand why it is dropping the event. When I parse the JSON in a linter, it is valid, if I try to make it in multiple lines instead of one, it goes to ES(logstash) but i creates an event per line. Any help appreciated. ---
**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\ **Post date:** [August 21, 2018, 6:07pm UTC](https://discuss.elastic.co/t/parsing-a-long-json-line-with-filebeat-to-logstash/145231/2 "2018-08-21T18:07:45Z")
@Gab_Ubi First, let's try to know if Filebeat is sending events to Logstash. If you replace your logstash configuration by the following: ```auto input { tcp { port => 5000 } beats { codec => "json" port => 5151 } } filter { json { source => "message" } } output { stdout { codec => rubydebug } } ``` And start Logstash interactively with the above configuration, `logstash -log.level debug -f ./myconfig.conf`, after start filebeat, do you see any outputs? ---
**Author:** ![Gab\_Ubi](https://avatars.discourse-cdn.com/v4/letter/g/b77776/32.png) [@Gab\_Ubi](https://discuss.elastic.co/u/Gab_Ubi)\ **Post date:** [August 21, 2018, 6:20pm UTC](https://discuss.elastic.co/t/parsing-a-long-json-line-with-filebeat-to-logstash/145231/3 "2018-08-21T18:20:44Z")
Hi Pier, thanks for your answer. I does send logs (I've monitored with TCPdump and watched the index getting created) but only when the json in not on a single line. When its on a single line, filebeat just seems to drop it. `,"pipeline":{"clients":1,"events":{"active":0,"filtered":1,"total":1}}}` ---
**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\ **Post date:** [August 21, 2018, 6:22pm UTC](https://discuss.elastic.co/t/parsing-a-long-json-line-with-filebeat-to-logstash/145231/4 "2018-08-21T18:22:36Z")
@Gab_Ubi Oh, does the single line in your log terminate with a `\n`? Filebeat looks for them to know when a line is complete, when it doesn't exist it assume there is still data that need to be appended. All loggers add a new line when a log statement is complete. ---
**Author:** ![Gab\_Ubi](https://avatars.discourse-cdn.com/v4/letter/g/b77776/32.png) [@Gab\_Ubi](https://discuss.elastic.co/u/Gab_Ubi)\ **Post date:** [August 21, 2018, 6:25pm UTC](https://discuss.elastic.co/t/parsing-a-long-json-line-with-filebeat-to-logstash/145231/5 "2018-08-21T18:25:00Z")
@pierhugues Good I'm going to try that and let you know 🙂 ---
**Author:** ![Gab\_Ubi](https://avatars.discourse-cdn.com/v4/letter/g/b77776/32.png) [@Gab\_Ubi](https://discuss.elastic.co/u/Gab_Ubi)\ **Post date:** [August 27, 2018, 7:07pm UTC](https://discuss.elastic.co/t/parsing-a-long-json-line-with-filebeat-to-logstash/145231/6 "2018-08-27T19:07:37Z")
You were exactly correct, i added \n at the end and its feeding now. Thank you! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 24, 2018, 7:07pm UTC](https://discuss.elastic.co/t/parsing-a-long-json-line-with-filebeat-to-logstash/145231/7 "2018-09-24T19:07:38Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.