# Logstash Filter Json To Array of Json
**URL:** https://discuss.elastic.co/t/logstash-filter-json-to-array-of-json/181345
**Category:** Logstash
**Created:** [May 16, 2019, 9:24am UTC](https://discuss.elastic.co/t/logstash-filter-json-to-array-of-json/181345 "2019-05-16T09:24:00Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@Vineethpp](https://discuss.elastic.co/u/Vineethpp)
#### Post date: [May 16, 2019, 9:24am UTC](https://discuss.elastic.co/t/logstash-filter-json-to-array-of-json/181345/1 "2019-05-16T09:24:00Z")
Hi,
I was trying get a logstash filter which will write json object to kafka as array of json object.
As of now with out any filter its writing the json object, but for me i want the kafka events as array of json objects (so that consumer understands it and reads).
this what it is happening now
{"http.method":"GET","http.path":"/serviceB"},"timestamp":1557911843380103,"parentId":"32f100cf25cd47ad","id":"1b61c2205ff61f8f","name":"get","localEndpoint":{"ipv4":"127.0.0.1","serviceName":"sample"},"traceId":"32f100cf25cd47ad"}
But i need it to be with in an array
[{"http.method":"GET","http.path":"/serviceB"},"timestamp":1557911843380103,"parentId":"32f100cf25cd47ad","id":"1b61c2205ff61f8f","name":"get","localEndpoint":{"ipv4":"127.0.0.0","serviceName":"sample"},"traceId":"32f100cf25cd47ad"}]
My filter
input {
http {
port =\> 9200
host =\> "0.0.0.0"
}
}
filter {
mutate {
remove\_field =\> ["headers", "@timestamp", "host", "@version"]
```
}
```
}
output {
kafka {
codec =\> json{}
bootstrap\_servers =\> "[kafka-service.test.com:9092](http://kafka-service.test.com:9092)"
topic\_id =\> "test"
}
}
Help me if anyone has any approach.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 16, 2019, 1:06pm UTC](https://discuss.elastic.co/t/logstash-filter-json-to-array-of-json/181345/2 "2019-05-16T13:06:22Z")
> [@Vineethpp](#):
>
> {"http.method":"GET","http.path":"/serviceB"},"timestamp":1557911843380103,"parentId":"32f100cf25cd47ad","id":"1b61c2205ff61f8f","name":"get","localEndpoint":{"ipv4":"127.0.0.1","serviceName":"sample"},"traceId":"32f100cf25cd47ad"}
I am not understanding what this represents. Can you configure
```
output { stdout { codec = > rubydebug } }
```
and show us what an event looks like?
---
### Author:  [@Vineethpp](https://discuss.elastic.co/u/Vineethpp)
#### Post date: [May 16, 2019, 1:20pm UTC](https://discuss.elastic.co/t/logstash-filter-json-to-array-of-json/181345/3 "2019-05-16T13:20:35Z")
We have an application which sends the data over http (body is json object ) like below.
I have written the logstash config to listen fof this http request, accept it , one filter is put there to remove the fields which is added by the input http plugin. Then output the json object to kafka topic. Consumers will read it from kafka topic, the problem here is the consumers are reading only array of json object, not the json object as is.. With the current logstash config, am seeing the messages/events in kafka topic is (the same as the application sends, this is trace data generated by zipkin instrumentation in that application)
{"http.method":"GET","http.path":"/serviceB"},"timestamp":1557911843380103,"parentId":"32f100cf25cd47ad","id":"1b61c2205ff61f8f","name":"get","localEndpoint":{"ipv4":"127.0.0.1","serviceName":"sample"},"traceId":"32f100cf25cd47ad"}
I want the message to be enclosed in square brackets in kafka topic, so that consumer can understand and consume it.
like this
[{"http.method":"GET","http.path":"/serviceB"},"timestamp":1557911843380103,"parentId":"32f100cf25cd47ad","id":"1b61c2205ff61f8f","name":"get","localEndpoint":{"ipv4":"127.0.0.1","serviceName":"sample"},"traceId":"32f100cf25cd47ad"}]
Am looking for a filter to add square bracket to the entire json object which comes out of the http input plugin and then write to kafka.
---
### Author:  [@Vineethpp](https://discuss.elastic.co/u/Vineethpp)
#### Post date: [May 16, 2019, 1:38pm UTC](https://discuss.elastic.co/t/logstash-filter-json-to-array-of-json/181345/4 "2019-05-16T13:38:06Z")
> [@Badger](#):
>
> output { stdout { codec = \> rubydebug } }
I did output { stdout { codec = \> rubydebug } }, the output looks like below, i need to add square brackets around it , i mean need to put the json object into an array of json object.
{
"timestamp" =\> 1557911261541631,
"id" =\> "97e9096d50474d6a",
"kind" =\> "SERVER",
"tags" =\> {
"mvc.controller.method" =\> "callService",
"mvc.controller.class" =\> "TracingController",
"http.path" =\> "/serviceA",
"http.method" =\> "GET"
},
"duration" =\> 17579,
"name" =\> "get",
"traceId" =\> "97e9096d50474d6a",
"localEndpoint" =\> {
"serviceName" =\> "sample",
"ipv4" =\> "127.0.0.1"
},
"remoteEndpoint" =\> {
"port" =\> 56208,
"ipv6" =\> "::1"
}
}
---
### Author:  [@pastechecker](https://discuss.elastic.co/u/pastechecker)
#### Post date: [May 16, 2019, 1:51pm UTC](https://discuss.elastic.co/t/logstash-filter-json-to-array-of-json/181345/5 "2019-05-16T13:51:28Z")
> [@Vineethpp](#):
>
> {"http.method":"GET","http.path":"/serviceB"},"timestamp":1557911843380103,"parentId":"32f100cf25cd47ad","id":"1b61c2205ff61f8f","name":"get","localEndpoint":{"ipv4":"127.0.0.1","serviceName":"sample"},"traceId":"32f100cf25cd47ad"}
Would that work: if you will place a value in a field?
{
"field" =\> "[{"http.method":"GET","http.path":"/serviceB"},"timestamp":1557911843380103,"parentId":"32f100cf25cd47ad","id":"1b61c2205ff61f8f","name":"get","localEndpoint":{"ipv4":"127.0.0.1","serviceName":"sample"},"traceId":"32f100cf25cd47ad"}]",
"@version" =\> "1",
"message" =\> "{"http.method":"GET","http.path":"/serviceB"},"timestamp":1557911843380103,"parentId":"32f100cf25cd47ad","id":"1b61c2205ff61f8f","name":"get","localEndpoint":{"ipv4":"127.0.0.1","serviceName":"sample"},"traceId":"32f100cf25cd47ad"}",
"host" =\> "testnode",
"@timestamp" =\> 2019-05-16T13:50:51.941Z
}
?
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 16, 2019, 1:56pm UTC](https://discuss.elastic.co/t/logstash-filter-json-to-array-of-json/181345/6 "2019-05-16T13:56:39Z")
OK. You could do it using json\_encode (which you will need to install). First copy all the interesting fields to another field, which we can put in [@metadata]
```
mutate { remove_field => ["@timestamp", "@version", "host", "message", "sequence"] }
ruby {
code => '
event.to_hash.each { |k,v|
event.set("[@metadata][fields][#{k}]", v)
}
'
}
```
Then encode it into another field, which again can be in [@metadata]
```
json_encode { source => "[@metadata][fields]" target => "[@metadata][string]" }
```
and output it using a plain codec
```
output { stdout { codec => plain { format => "[%{[@metadata][string]} ]
" } } }
```
Note that you use a literal newline in the format string to tell it to append a newline to the output.
---
### Author:  [@Vineethpp](https://discuss.elastic.co/u/Vineethpp)
#### Post date: [May 16, 2019, 2:51pm UTC](https://discuss.elastic.co/t/logstash-filter-json-to-array-of-json/181345/8 "2019-05-16T14:51:35Z")
Thank you @Badger It worked like a charm.
But i think logstash should have a json object to array of json object converter.
---
### Author:  [@Vineethpp](https://discuss.elastic.co/u/Vineethpp)
#### Post date: [May 16, 2019, 2:52pm UTC](https://discuss.elastic.co/t/logstash-filter-json-to-array-of-json/181345/9 "2019-05-16T14:52:24Z")
Thank you @pastechecker I tried this approach, didnt worked actually. But the below mentioned approach by @Badger worked.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 13, 2019, 2:52pm UTC](https://discuss.elastic.co/t/logstash-filter-json-to-array-of-json/181345/10 "2019-06-13T14:52:28Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.