Where Metricbeat stores collected data

Elastic Stack Beats
1

Hello,

while I was testing functionalists of Metricbeat following question came up on my mind. If the output is unreachable, can this cause partition overflow? Based on performed tests my conclusion was that data is stored locally during the unavailable output because as soon it became online all data was ingested to Elasticsaerch.

BR,
Mladen

2

Metricbeat stores messages in its internal queue. By default it's an in-memory queue. It's configured to store 4096 messages. But you can change it to fit your environment.
Of course, this can cause overflows is you configure the internal queue. But if you use the defaults, I doubt that a regular server would fail.

You can read more about internal queues here: https://www.elastic.co/guide/en/beats/metricbeat/current/configuring-internal-queue.html
Also, spooling to disk is going to be introduced in 6.3. If you want Metricbeat to write its queue to disk I recommend you to read this and use it if possible: https://www.elastic.co/guide/en/beats/metricbeat/master/configuring-internal-queue.html#configuration-internal-queue-spool

3

@kvch thank you for this explanation. You help me a lot to understand the hole process.

BR,
Mladen

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.