# Logstash http output plugin customize building json array for batching
**URL:** https://discuss.elastic.co/t/logstash-http-output-plugin-customize-building-json-array-for-batching/275535
**Category:** Logstash
**Created:** [June 10, 2021, 8:24am UTC](https://discuss.elastic.co/t/logstash-http-output-plugin-customize-building-json-array-for-batching/275535 "2021-06-10T08:24:46Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@RahulGS](https://discuss.elastic.co/u/RahulGS)
#### Post date: [June 10, 2021, 8:24am UTC](https://discuss.elastic.co/t/logstash-http-output-plugin-customize-building-json-array-for-batching/275535/1 "2021-06-10T08:24:46Z")
Hello ,
We have a pipeline running on logstash 6.6.2 with input(redis)-\>filter(ruby)-\>output(http)
how do you batch http output using custom format ? we can't use format="json\_batch" because it will construct a simple json array which will be invalid per the schema defined by the http endpoint (owned by different team).
How do i go about combining batch of messages using my own json format ? any ideas/suggestions ?
Example:
Input log messages :
```auto
{json1} , {json2} , {json3} .. {jsonN}
```
output (with format="json\_batch"):
```auto
[{json1} , {json2} , {json3} .. {jsonN}]
```
output (customized):
```auto
{ records : [{json1} , {json2} , {json3} .. {jsonN}] }
```
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [June 10, 2021, 4:29pm UTC](https://discuss.elastic.co/t/logstash-http-output-plugin-customize-building-json-array-for-batching/275535/2 "2021-06-10T16:29:31Z")
> [@RahulGS](#):
>
> How do i go about combining batch of messages using my own json format ?
That is not simple. outputs sometimes have a receive method that consumes a single event, and sometimes have a multi\_receive method that consumes an array of events (a batch). The multi\_receive method typically iterates over the array and processes the events one at a time. The http output is unusual in that it can process the entire array in one shot. However, it provides no flexibility on the request format that it sends over http.
That means you will have to do some work in the filter section, and the filter section does not see a batch, it sees individual events.
If you are OK with sending one event per request
```auto
{ records : [{json1}] }
{ records : [{json2}] }
{ records : [{json3}] }
```
etc., then modifying the format is not very hard.
Otherwise you are going to have to aggregate the events to create a batch. That can be done. There is an example of combining events with a limit on size [here](https://discuss.elastic.co/t/aggregate-filter-push-on-event-size-limit/180875/2). You could change that to add a set number of events to an array, effectively re-creating the batch.
Which approach do you prefer?
---
### Author:  [@RahulGS](https://discuss.elastic.co/u/RahulGS)
#### Post date: [June 10, 2021, 4:41pm UTC](https://discuss.elastic.co/t/logstash-http-output-plugin-customize-building-json-array-for-batching/275535/3 "2021-06-10T16:41:34Z")
Thanks Badger. Since I am trying to optimize the number of HTTP Calls , anything that i can do to send batch of messages in a single request would help.
In your example , filter "aggregate" seems to need taskid to group events , where as the log messages i have do not have any correlation between them . how can we use it ?
---
### Author:  [@RahulGS](https://discuss.elastic.co/u/RahulGS)
#### Post date: [June 10, 2021, 5:07pm UTC](https://discuss.elastic.co/t/logstash-http-output-plugin-customize-building-json-array-for-batching/275535/4 "2021-06-10T17:07:55Z")
i see what you are saying , basically consider all messages as part of the same taskid and create batches either based on the size or number of messages. That should work 🙂
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [June 10, 2021, 5:08pm UTC](https://discuss.elastic.co/t/logstash-http-output-plugin-customize-building-json-array-for-batching/275535/5 "2021-06-10T17:08:53Z")
Assuming you do not care which events are combined with which others then you can add one
```
mutate { add_field => { "[@metadata][task]" => "1" } }
```
and then use
```
task_id => "[@metadata][task]"
```
Normally you have to set pipeline.workers to 1 to use aggregate but in this case it may work even if you do not.
---
### Author:  [@RahulGS](https://discuss.elastic.co/u/RahulGS)
#### Post date: [June 14, 2021, 1:54am UTC](https://discuss.elastic.co/t/logstash-http-output-plugin-customize-building-json-array-for-batching/275535/6 "2021-06-14T01:54:37Z")
Hi Badger ,
i am able to aggregate and batch the events to output successfully , however looks like the timeout functionality is not working . i have set the following in the first aggregate filter.
```auto
push_map_as_event_on_timeout => true
timeout => 10
timeout_code => "event.set('[@metadata][timeToFlush]', true)"
timeout_task_id_field' => "[@metadata][taskid]"
```
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [June 14, 2021, 2:30am UTC](https://discuss.elastic.co/t/logstash-http-output-plugin-customize-building-json-array-for-batching/275535/7 "2021-06-14T02:30:29Z")
I would suggest removing the drop {} and replacing the output with
```
stdout { codec => rubydebug { metadata => true } }
```
output. See if the timeout event shows up.
---
### Author:  [@RahulGS](https://discuss.elastic.co/u/RahulGS)
#### Post date: [June 14, 2021, 5:59am UTC](https://discuss.elastic.co/t/logstash-http-output-plugin-customize-building-json-array-for-batching/275535/8 "2021-06-14T05:59:10Z")
Thanks Badger , I had some output conditions based on original event properties , for timeout scenarios those were not set , once that was fixed , everything worked 🙂
---
### Author:  [@RahulGS](https://discuss.elastic.co/u/RahulGS)
#### Post date: [June 15, 2021, 4:11am UTC](https://discuss.elastic.co/t/logstash-http-output-plugin-customize-building-json-array-for-batching/275535/9 "2021-06-15T04:11:17Z")
it looks like sometimes i am getting individual events at the output even though all of them should be aggregated or timed out events . is it expected ? [as any raw event should have been dropped at filter] . is this because of some kind of concurrency issue ?
i was also able to confirm these individual events that reach output were also output as part of aggregated events.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 13, 2021, 4:11am UTC](https://discuss.elastic.co/t/logstash-http-output-plugin-customize-building-json-array-for-batching/275535/10 "2021-07-13T04:11:49Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.