# Can we delay output to Elasticsearch in batch?

**URL:** <https://discuss.elastic.co/t/can-we-delay-output-to-elasticsearch-in-batch/317137>\
**Category:** Elastic Agent\
**Created:** [October 20, 2022, 3:39pm UTC](https://discuss.elastic.co/t/can-we-delay-output-to-elasticsearch-in-batch/317137 "2022-10-20T15:39:13Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![quentin.legraverend](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/quentin.legraverend/32/66373_2.png) [@quentin.legraverend](https://discuss.elastic.co/u/quentin.legraverend)\
**Post date:** [October 20, 2022, 3:39pm UTC](https://discuss.elastic.co/t/can-we-delay-output-to-elasticsearch-in-batch/317137/1 "2022-10-20T15:39:13Z")

</div>

Hello,

I have an Elastic stack self-managed in the cloud with a Fleet Server on it.  
I have a second environment on-premise with a really limited bandwidth and I want to have some Elastic Agents here also.

My first idea was to deploy a Fleet Server on-premise that will grab all flows from on-premise Elastic Agents and then forward those data to the Elastic stack in the cloud, but I can not find any parameter to delay Fleet Server communication to Elasticsearch in batches (like 1 batch every hour for example).

So my 2 questions are:

- Is the Fleet Server capable of sending data in batch mode to Elasticsearch?
- If not, can an enrolled Elastic Agent do it by itself?

Thank you in advance!  
Quentin

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [October 20, 2022, 10:11pm UTC](https://discuss.elastic.co/t/can-we-delay-output-to-elasticsearch-in-batch/317137/2 "2022-10-20T22:11:06Z")

</div>

Check [documentation](https://www.elastic.co/guide/en/fleet/current/elasticsearch-output.html#output-elasticsearch-performance-tuning-settings) and [this](https://www.elastic.co/guide/en/beats/filebeat/current/configuring-internal-queue.html). Test how will be if you change params:

- bulk\_max\_size
- increase compression\_level
- backoff.max

---

<div class="post-metadata">

**Author:** ![quentin.legraverend](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/quentin.legraverend/32/66373_2.png) [@quentin.legraverend](https://discuss.elastic.co/u/quentin.legraverend)\
**Post date:** [October 24, 2022, 1:57pm UTC](https://discuss.elastic.co/t/can-we-delay-output-to-elasticsearch-in-batch/317137/3 "2022-10-24T13:57:07Z")

</div>

Thank you for your answer.

I hadn't though about these parameters. I gave these a try even if this is more a workaround.  
`bulk_max_size` & `backoff.max` do not really help in my case, but it looks like `compression_level` helps a bit. CPU usage is currently not a bottleneck for me so I guess I can safely use this parameter for now. I will may be try to use it in conjunction with QoS at router level. That should be sufficient to preserve upload bandwidth.

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [October 24, 2022, 7:46pm UTC](https://discuss.elastic.co/t/can-we-delay-output-to-elasticsearch-in-batch/317137/4 "2022-10-24T19:46:20Z")

</div>

FB has more tuning settings, if is feasible, test it with speed params. Check [doc](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-reference-yml.html).  
Also if is possible, do not use HTTPS. HTTP is simpler.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 21, 2022, 7:47pm UTC](https://discuss.elastic.co/t/can-we-delay-output-to-elasticsearch-in-batch/317137/5 "2022-11-21T19:47:06Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
