Metricbeat Storage in ElasticSearch index

[benjamin.watson]

Greetings all,

I'm designing a monitoring and alerting platform and intend to base it largely on ElasticSearch and leverage Metricbeat for per host metrics collection. I've looked up some general (and often conflicting) ElasticSearch "best practices" when it comes to cluster, index, and shard configuration. And I know the issue I'm facing is the general answer "it depends". It depends on how much data you want to keep, how hot/warm/cold you want it, if you'll do roll-up indices, which metricbeat modules and metricsets you will use, etc.

Given that, I still need to start planning and get an initial cluster sized "the best I can". Assuming I know the number of hosts (machines/VMs) I want to monitor via metricbeat, and assuming I use the default out-of-the-box configuration for the system module, approximately how much data can I expect to ingest into ElasticSearch per monitored host per hour (or day, or week)?

For example, assume my default metricbeat config looks like this:

metricbeat.modules:
                - module: system
                  metricsets:
                    - cpu
                    - load
                    - memory
                    - network
                    - process
                    - process_summary
                    # - uptime
                    # - core
                    # - diskio
                    # - filesystem
                    # - fsstat
                    # - raid
                    # - socket
                  enabled: true
                  period: 10s
                  processes: ['.*']

How much ES index storage does that translate to on a per-host basis? I know there is a lot of fine tuning that can be done (e.g. filtering top_N processes, etc., etc.). But what does the above "default" configuration typically yield (assuming ES/Beats 7.x).

Are there any "rules of thumb" guides out there that generalize the storage volume generated by the various metricbeat modules/configurations?

Thanks!

Ben

[benjamin.watson]

MetricBeat Stats

The following table summarizes some experiments running the metricbeat agent on
various platforms and with various configurations to determine the amount of
storage within ElasticSearch to store the metrics over periods of time. The
idea here is to get a per-monitored-host storage requirement so we can
appropriately size an Elastic cluster to serve as an infrastructure monitoring
platform.

Stats Collection Methodology

Since I already use Ansible to configuration manage hosts and have playbooks for
managing ElasticSearch, I employed them to spin up a new VM host with
ElasticSearch. The node was configured as a master/ingest/data node. The VM
also has Kibana installed to allow me to easily observe index creation and
document arrival into the index.

Once initially provisioned, we have ElasticSearch with no indicies ready to
receive some data. I then used Ansible to provision the metricbeat agent to
a host and configure the metricbeat modules and metricsets. Next, I run the
metricbeat agent for 5 minutes and observe the created index size in Kibana. I
take note of the number of documents and size of the index. I record this
information into a spreadsheet, where simple cell-based multiplcation calculates
the extrapolated storage for 1 hour, 1 day, 1 month, and 1 year increments.

After each 5-minute run, I delete the index, reconfigure metricbeat, and run the
agent for another 5-minute run. I performed this with metricbeat running on
Linux (the same node running ElasticSearch/Kibana) and with metricbeat running
on Windows.

Results

Platform	Modules	Period (s)	# of Docs (5m)	Size (MB) (5m)	# of Docs (1hr)	Size (MB) (1hr)	# of Docs (1d)	Size (MB) (1d)	# of Docs (1mo)	Size (MB) (1mo)	Size (GB) (1mo)	# of Docs (1y)	Size (MB) (1y)	Size (GB) (1y)
Linux	system	5	7617	6.1	91404	73.2	2.1937e+06	1756.8	6.58109e+07	52704	51.4688	8.00699e+08	641232	626.203
Linux	system	10	3673	3	44076	36	1.05782e+06	864	3.17347e+07	25920	25.3125	3.86106e+08	315360	307.969
Linux	system	20	1838	1.5	22056	18	529344	432	1.58803e+07	12960	12.6562	1.93211e+08	157680	153.984
Linux	system	10	3828	3.2	45936	38.4	1.10246e+06	921.6	3.30739e+07	27648	27	4.02399e+08	336384	328.5
Linux	system	10	3937	3.2	47244	38.4	1.13386e+06	921.6	3.40157e+07	27648	27	4.13857e+08	336384	328.5
Linux	system	10	4206	3.3	50472	39.6	1.21133e+06	950.4	3.63398e+07	28512	27.8438	4.42135e+08	346896	338.766
Linux	system	10	4271	3.4	51252	40.8	1.23005e+06	979.2	3.69014e+07	29376	28.6875	4.48968e+08	357408	349.031
Linux	system	10	4173	3.3	50076	39.6	1.20182e+06	950.4	3.60547e+07	28512	27.8438	4.38666e+08	346896	338.766
Windows	windows	10	5790	1	69480	12	1.66752e+06	288	5.00256e+07	8640	8.4375	6.08645e+08	105120	102.656
Windows	windows	10	444	0.12	5328	1.44	127872	34.56	3.83616e+06	1036.8	1.0125	4.66733e+07	12614.4	12.3187

The first 3 rows correspond to a "default" metricbeat configuration, where the
only variation is the metricbeat reporting period. The default reporting period
is 10 seconds (row 2), but I also experimented with cutting that time in half
for more resolution (row 1) and doubling that time for storage considerations
(row 3). The configuration below is representative of these cases.

logging:
  files: {keepfiles: 2, name: metricbeat.log, path: /var/log/}
  level: warning
  to_files: true
  to_syslog: false
metricbeat.modules:
- enabled: true
  metricsets: [cpu, load, memory, network, process, process_summary]
  module: system
  period: 10s
  processes: [.*]
output:
  elasticsearch:
    enabled: true
    hosts: ['http://elastic.example.com:9200']
    index: metricbeat-%{[beat.version]}-default
    password: changeme
    username: elastic
setup:
  dashboards: {enabled: true}
  kibana: {host: 'http://kibana.example.com:5601'}
  template: {name: 'metricbeat-%{[beat.version]}', pattern: 'metricbeat-%{[beat.version]}-*'}

The next 5 rows correspond to a default 10-second reporting period, but for each
row I added an additional metricset. I incrementally added the uptime, core,
diskio, filesystem, and fsstat metricsets.

The last 2 rows correspond to metricbeat running on Windows. The first Windows
row is the metricbeat windows module collecting the service metricset. The
configuration looks like this:

logging:
    files:
        keepfiles: 2
        name: metricbeat.log
        path: C:\Metricbeat\logs
    level: debug
    to_files: true
    to_syslog: false
metricbeat.modules:
-   enabled: true
    metricsets:
    - service
    module: windows
    period: 10s
output:
    elasticsearch:
        enabled: true
        hosts:
        - http://elastic.example.com:9200
        index: metricbeat-%{[beat.version]}
        password: changeme
        username: elastic
setup:
    template:
        name: metricbeat-%{[beat.version]}
        pattern: metricbeat-%{[beat.version]}-*

The last Windows row uses the (still beta) perfmon metricset and is configured
like this:

logging:
    files:
        keepfiles: 2
        name: metricbeat.log
        path: C:\Metricbeat\logs
    level: debug
    to_files: true
    to_syslog: false
metricbeat.modules:
-   enabled: true
    metricsets:
    - perfmon
    module: windows
    perfmon.counters:
    -   instance_label: processor_name
        instance_name: total
        measurement_label: processor.time.total.pct
        query: \Processor(_Total)\% Processor Time
    -   instance_label: physical_disk.name
        instance_name: total
        measurement_label: physical_disk.time.total.pct
        query: \PhysicalDisk(_Total)\% Disk Time
    -   instance_label: physical_disk.name
        instance_name: total
        measurement_label: physical_disk.time.read.pct
        query: \PhysicalDisk(_Total)\% Disk Read Time
    -   instance_label: physical_disk.name
        instance_name: total
        measurement_label: physical_disk.time.write.pct
        query: \PhysicalDisk(_Total)\% Disk Write Time
    -   instance_label: logical_disk.name
        instance_name: total
        measurement_label: logical_disk.space.free.pct
        query: \LogicalDisk(_Total)\% Free Space
    -   instance_label: logical_disk.name
        instance_name: total
        measurement_label: logical_disk.space.free.mb
        query: \LogicalDisk(_Total)\Free Megabytes
    -   instance_label: paging_file.name
        instance_name: total
        measurement_label: paging_file.usage.pct
        query: \Paging File(_Total)\% Usage
    -   instance_label: memory.name
        instance_name: total
        measurement_label: memory.available.mbytes
        query: \Memory()\Available MBytes
    -   instance_label: numa_node.name
        instance_name: total
        measurement_label: numa_node.memory.available.mbytes
        query: \NUMA Node Memory(_Total)\Available MBytes
    -   instance_label: numa_node.name
        instance_name: total
        measurement_label: numa_node.memory.total.mbytes
        query: \NUMA Node Memory(_Total)\Total MBytes
    -   instance_label: system_info.name
        instance_name: total
        measurement_label: system.processes.count
        query: \System()\Processes
    -   instance_label: system_info.name
        instance_name: total
        measurement_label: system.threads.count
        query: \System()\Threads
    -   instance_label: system_info.name
        instance_name: total
        measurement_label: system.uptime.seconds
        query: \System()\System Up Time
    -   instance_label: network_interface.name
        instance_name: vmxnet3
        measurement_label: network.interface.in.bytes
        query: \Network Interface(vmxnet3 Ethernet Adapter)\Bytes Received/sec
    -   instance_label: network_interface.name
        instance_name: vmxnet3
        measurement_label: network.interface.out.bytes
        query: \Network Interface(vmxnet3 Ethernet Adapter)\Bytes Sent/sec
    perfmon.group_measurements_by_instance: true
    perfmon.ignore_non_existent_counters: true
    period: 10s
output:
    elasticsearch:
        enabled: true
        hosts:
        - http://elastic.example.com:9200
        index: metricbeat-%{[beat.version]}
        password: changeme
        username: elastic
setup:
    template:
        name: metricbeat-%{[beat.version]}
        pattern: metricbeat-%{[beat.version]}-*

Conclusions

Metricbeat is just once piece of the holistic Application and Infrastructure
monitoring solution. We'll need this metric information along with logging and
application performance (APM) data to paint a good picture of the healt and
status of applications and infrastructure, and to alarm/alert when the data
tells us something is abnormal.

Running metricbeat on Linux and collecting all but the raid and socket
metricsets at a 10-second collection period account for nearly 1GB of data per
day per monitored host. For Windows, it is currently about 300MB of data per
day per monitored host. If you have 1000+ hosts to monitor, this could easily
top 1TB per day. And again, this is not accounting for other data sources such
as logs, APM telemetry, etc. Furthermore, this isn't taking into account
ElasticSearch data high availability with multiple primaries and replicas keeping
the data resilient/accessible.

The moral of the story here is you will need to clearly define the hosts you
want to monitor and come up with some sort of data retention strategy. You'll
likely need to make use of Curator and/or ILM policies to ensure your not
hanging on to perishable data for too long. Another strategy may be to make
use of rollup indexes to summarize metric data over time.

I'm certainly open to hearing stories from the community on how they've used
ElasticSearch as the heart of an Application and Infrastructure monitoring platform
and the strategies they've employed to balance functionality and resource
utilization.