Metricbeat 5.5 sending events to Logstash in batches

Elastic Stack Beats
1

Is it possible for Metricbeat to send multiple events to Logstash in batches?

For example, collect system CPU and Memory metrics every second, but ship the metrics to Logstash in batches (e.g. 60 events) every minute? Thanks.

Sample config:

metricbeat.modules:
- module: system
  metricsets:
    - cpu
    - memory
  enabled: true
  period: 1s
  processes: ['.*']

output.logstash:
  hosts: ["localhost:5044"]
2

Why do you want to do this? I believe the outputs support setting the flush_interval (default is 1s). You could set this to 60s then internally queued events would be sent every 60s or when the queue reaches it max size (configured with bulk_max_size and defaults to 2048).

3

Thanks Andrew. This is actually what I was looking for. I didn't know the "flush_interval" setting exist for the logstash output.

4

@andrewkroh, will Metricbeat send the internally queued events to Logstash as a single request? Thanks.

5

It should send the events in a batch. The batch size may start small and grow once it determines the Logstash is able to handle the larger batch sizes (this behavior was disabled in 6.0).

6

Does Metricbeat average out the metrics for CPU, Memory, Network, etc. over the period (e.g. 10s) set in the config file? Or does it collect raw metrics at the time? Thanks.

For example, collecting every 10s:

- module: system
  metricsets:
    - cpu
    - memory
    - network
  enabled: true
  period: 10s
  processes: ['.*']
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.