Is Metricbeat meant to ship hundreds of docs per second?

Elastic Stack Beats
#1

Hiii All,

I've just started adding Metricbeat to my K8S cluster following this tutorial: https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-kubernetes.html

Everything works fine and I was able to see Metricbeat docs being received at another ELK stack instance. The confusion I had was the documents seem to be too many --- after the first night, there has been already over 2 million metric docs which takes nearly 1Gb storage... i wondered whether that's an expected behavior but I couldn't find any posts / official documentation -mentioning how frequent the Metricbeat sends metrics....

#2

Welcome to our community! :smiley:

I guess that really depends on how much you are collecting from. How many pods/containers?

#3

Thanks for replying! This is my K8S YAML created following the tutorial, with some minor modification to fit my deployment needs. I am running a cluster of 6 nodes on Azure cloud, and 34 pods in total (excluding metricbeat pods)

#5

Bump + 1

Anyone can help with this?

#7

Yes it can be that much.

You have 34 Pods , so that is about 29mb each per day.

If you look at the default yml which you probably loaded.

https://raw.githubusercontent.com/elastic/beats/7.9/deploy/kubernetes/metricbeat-kubernetes.yaml

You can see many metrics are collected at a 10 second interval.

You can greatly affect how many docs how much storage is required by tuning which metrics that are collected and the interval they are collected.

That yml config is a default.

Example change the collection rate to 1min and you will reduce the docs and storage by a factor of 6.

You should think about what you want to collect and how often.

That said you can also roll-up the data to reduce the docs and storage as the data aged.

https://www.elastic.co/guide/en/kibana/7.9/data-rollups.html

2 Likes
#8

Thanks for the tips!!