# Delay in beats documents to get stored into Elasticsearch

**URL:** https://discuss.elastic.co/t/delay-in-beats-documents-to-get-stored-into-elasticsearch/189947
**Category:** Elasticsearch
**Created:** [July 11, 2019, 9:03am UTC](https://discuss.elastic.co/t/delay-in-beats-documents-to-get-stored-into-elasticsearch/189947 "2019-07-11T09:03:07Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![vipulnewaskar7](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vipulnewaskar7/32/47332_2.png) [@vipulnewaskar7](https://discuss.elastic.co/u/vipulnewaskar7)
#### Post date: [July 11, 2019, 9:03am UTC](https://discuss.elastic.co/t/delay-in-beats-documents-to-get-stored-into-elasticsearch/189947/1 "2019-07-11T09:03:08Z")

</div>

Hello Guys,

I am having a monitoring stack, in which, I am sending beats data directly to elasticsearch without logstash in between.  
Data sent by beats is not get stored in elasticsearch immediately. It lags ~4-5 Min, and lag increases further after certain period.  
ElasticSearch Version: 6.7  
I am not using for prod environment, hence, I have kept  
only 1 master node, 1 data node and 1 client node.  
Master and Data Node have 2-2 Gi of RAM and 1 Gi of heap space.  
Client Node has 4 Gi RAM and 2 Gi heap space.

\*I know this isn't a recommended architecture, but, I am using it just for dev environment at non-critical level, and I have to use this in limited resources.

What changes I should do in architecture/configuration like flush\_time, queue\_size, etc.?

---

<div class="post-metadata">

### Author: ![vipulnewaskar7](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vipulnewaskar7/32/47332_2.png) [@vipulnewaskar7](https://discuss.elastic.co/u/vipulnewaskar7)
#### Post date: [July 11, 2019, 1:04pm UTC](https://discuss.elastic.co/t/delay-in-beats-documents-to-get-stored-into-elasticsearch/189947/2 "2019-07-11T13:04:51Z")

</div>

One thing I have observed is,  
When data is not reaching the elasticsearch,

- Beats keep data stored in their queues/spools
- when I restart elasticsearch client node, all of a sudden, data starts getting stored in the elasticsearch.

Just to give you an idea about the size of data,  
It's hardly 10000 Documents per minute, each document being 2-3 kb  
In short 20-30 MB per minute.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 8, 2019, 1:04pm UTC](https://discuss.elastic.co/t/delay-in-beats-documents-to-get-stored-into-elasticsearch/189947/3 "2019-08-08T13:04:54Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
