Delay in beats documents to get stored into Elasticsearch

Elastic Stack Elasticsearch
1

Hello Guys,

I am having a monitoring stack, in which, I am sending beats data directly to elasticsearch without logstash in between.
Data sent by beats is not get stored in elasticsearch immediately. It lags ~4-5 Min, and lag increases further after certain period.
ElasticSearch Version: 6.7
I am not using for prod environment, hence, I have kept
only 1 master node, 1 data node and 1 client node.
Master and Data Node have 2-2 Gi of RAM and 1 Gi of heap space.
Client Node has 4 Gi RAM and 2 Gi heap space.

*I know this isn't a recommended architecture, but, I am using it just for dev environment at non-critical level, and I have to use this in limited resources.

What changes I should do in architecture/configuration like flush_time, queue_size, etc.?

2

One thing I have observed is,
When data is not reaching the elasticsearch,

  • Beats keep data stored in their queues/spools
  • when I restart elasticsearch client node, all of a sudden, data starts getting stored in the elasticsearch.

Just to give you an idea about the size of data,
It's hardly 10000 Documents per minute, each document being 2-3 kb
In short 20-30 MB per minute.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.