How to store metricbeat data in local disk

Elastic Stack Beats
1

Hi All,

We had a discussion in meeting about the metricbeat data sending through elasticsearch output and if the output elasticsearch is went down where the metric data will store?

I was reading this article,

https://www.elastic.co/guide/en/beats/metricbeat/current/configuring-internal-queue.html

In that by default the metricbeat uses internal queue to store events in memory before publishing them. For our case we will be capturing metrics data and sending to elasticsearch for the period interval of every 5 mins.

What is the idle number I should defined in the events memory queue?
The below sample Configuration can store upto 4096 events if I want to store all the events what should I need to do?

Please correct me If my understanding is wrong.

queue.mem:
  events: 4096
  flush.min_events: 512
  flush.timeout: 5s

What will happen if I don't specify the no of events in the queue?

Will it store all the metrics data untill the output of elasticsearch comes up?

Also there is an another option in metricbeat called file spool queue where we can store all the events on the disk

The below sample configuration can store the events upto 512MB. Can I specifiy the size of 1GB ??
so that the events will be stored in the local and once the output of elasticsearch comes up all the data will be flushed to the elasticsearch based on the write buffer

queue.spool:
  file:
    path: "${path.data}/spool.dat"
    size: 512MiB
    page_size: 16KiB
  write:
    buffer_size: 10MiB
    flush.timeout: 5s
    flush.events: 1024

Please let me know your thoughts it would be helpful.

Regards,
Ganeshbabu R

2

Yes, you can define the size of the spool file. Just take into account that once the spool file is full, Metricbeat will start dropping new events.

You can do the math of what your retention period will be for a certain size, based on the traffic amount specific to your settings.

Best regards

3

Does the file spools queue works?

I have same concern, Metricbeat fail safe/over function on output.

But got an error during implementation

4

Can you please share your settings and the error?

5

It is on the thread.

kindly check this link

Metricbeat fail safe/over function on output - #8 by jogoinar10

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.