# How to parse logstash's data to a json array? **URL:** \ **Category:** Logstash\ **Created:** [December 10, 2020, 1:03pm UTC](https://discuss.elastic.co/t/how-to-parse-logstashs-data-to-a-json-array/258274 "2020-12-10T13:03:16Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![iooi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iooi/32/79231_2.png) [@iooi](https://discuss.elastic.co/u/iooi)\ **Post date:** [December 10, 2020, 1:03pm UTC](https://discuss.elastic.co/t/how-to-parse-logstashs-data-to-a-json-array/258274/1 "2020-12-10T13:03:16Z")
How to parse logstash's data to a json array? Using filebeat sending a file to logstash. The file includes 2 line of records: ```auto {"request_id": "m2ee22d045f6c5ce07fe43dbdaea1de0","method": "GET","status": "304","forwarded_for": "","host": "localhost","url": "/","referer": "","remote_ip": "10.0.2.2","server_ip": "10.0.2.15","user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_0_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36","latency": "0s","occurred_at": "2020-12-10T03:39:29+00:00"} {"request_id": "m2ee22d045f6c5ce07fe43dbdaea1de0","method": "GET","status": "304","forwarded_for": "","host": "localhost","url": "/","referer": "","remote_ip": "10.0.2.2","server_ip": "10.0.2.15","user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_0_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36","latency": "0s","occurred_at": "2020-12-10T03:39:29+00:00"} ``` filebeat.yml ```auto filebeat.inputs: - type: log paths: - /var/log/nginx.log processors: - decode_json_fields: fields: ["message"] process_array: true max_depth: 1 target: "" overwrite_keys: true add_error_key: false output.logstash: hosts: ["logstash:5044"] ``` logstash.conf ```auto input { beats { port => 5044 } } filter { mutate { remove_field => [ "agent", "event", "message" ... ] } } output { stdout { codec => rubydebug } } ``` Logstash got the data as ```auto { "latency" => "0s", "url" => "/", "occurred_at" => "2020-12-10T08:08:10+00:00", "server_ip" => "10.0.2.15", "host" => "localhost", "request_id" => "ede6f0f02934cd18bcab1f9f531586e2", "remote_ip" => "10.0.2.2", "forwarded_for" => "", "user_agent" => "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_0_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36", "status" => "304", "referer" => "", "method" => "GET" } { "latency" => "0s", "url" => "/", "occurred_at" => "2020-12-10T08:08:10+00:00", "server_ip" => "10.0.2.15", "host" => "localhost", "request_id" => "ede6f0f02934cd18bcab1f9f531586e2", "remote_ip" => "10.0.2.2", "forwarded_for" => "", "user_agent" => "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_0_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36", "status" => "304", "referer" => "", "method" => "GET" } ``` Want to get data as ```auto [ { "latency" => "0s", "url" => "/", "occurred_at" => "2020-12-10T08:08:10+00:00", "server_ip" => "10.0.2.15", "host" => "localhost", "request_id" => "ede6f0f02934cd18bcab1f9f531586e2", "remote_ip" => "10.0.2.2", "forwarded_for" => "", "user_agent" => "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_0_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36", "status" => "304", "referer" => "", "method" => "GET" }, { "latency" => "0s", "url" => "/", "occurred_at" => "2020-12-10T08:08:10+00:00", "server_ip" => "10.0.2.15", "host" => "localhost", "request_id" => "ede6f0f02934cd18bcab1f9f531586e2", "remote_ip" => "10.0.2.2", "forwarded_for" => "", "user_agent" => "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_0_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36", "status" => "304", "referer" => "", "method" => "GET" } ] ``` I researched both filebeat and logstash's documents. But don't know if make it in filebeat or logstash's `filter`. All the necessary data are in `message` only. Just want to rewrite this context. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [December 10, 2020, 5:34pm UTC](https://discuss.elastic.co/t/how-to-parse-logstashs-data-to-a-json-array/258274/2 "2020-12-10T17:34:02Z")
If you want to combine lines from the same file you could do that using a multiline configuration in filebeat. Alternatively, use an aggregate filter in logstash with the file path as the task id. Something similar to example 3 in the documentation. ---
**Author:** ![iooi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iooi/32/79231_2.png) [@iooi](https://discuss.elastic.co/u/iooi)\ **Post date:** [December 11, 2020, 2:08am UTC](https://discuss.elastic.co/t/how-to-parse-logstashs-data-to-a-json-array/258274/3 "2020-12-11T02:08:38Z")
Thank you. I tried add this to filebeat.yml ```auto filebeat.inputs: - input_type: log paths: - /var/log/nginx.log multiline.pattern: '^\{' multiline.negate: true multiline.match: after ``` It doesn't work. In logstash.conf, I added this to `filter` ```auto aggregate { task_id => "/var/log/nginx.log" } ``` Can't run logstash successfully. Do you mean this example? [Example #3 : no end event](https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html#plugins-filters-aggregate-example3) Could you give me an example for it? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 8, 2021, 2:08am UTC](https://discuss.elastic.co/t/how-to-parse-logstashs-data-to-a-json-array/258274/4 "2021-01-08T02:08:40Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.