ILM strategy for filebeat, metricbeat and apm indices

Elastic Stack Elasticsearch
1

Hi Team,
I have 1 node elasticsearch cluster, which has 60GB RAM , 1.2 TB disk. I have 1 filebeat and 1 metricbeat index each receiving 6 GB data per day and 1 apm index which is receiving 10GB data per day. I want to retain 60 days' data and delete 1st day's data on 61st day (for each index). If you could propose strategy for implementing this ILM, it would be really helpful. Also I have below questions:

  1. How much disk space I would require?
  2. What would be effect on CPU usage for this much data and do I need to increase RAM?
  3. How increasing shards for indices will help in this situation?

Thanks !

2

Hi @aksh_18!

Since your questions are not filebeat/metricbeat specific but mostly about Elasticsearch, could you please redirect them to Elasticsearch's topics so as to get help from the team there?

3

Welcome to our community! :smiley:

Please make sure you have read the Elasticsearch ILM documentation, as well as the relevant docs for both of your Beats. Once you've got that setup, configure your policy to match the required retention rate and you should be good to go!

4

Hi Team,
If I follow this approach , will it be correct for 2 node cluster with one HOT and 1 WARM node? Also is it possible to have HOT and WARM phases in the same node? ( Note: 24GB data is getting ingested per day and filebeat, metircbeat , apm-transaction and apm-span are having 12 indices each. So at a time 4 indices, one from each agent , will be in HOT phase)

IndicesFile
IndicesFile1087×103 3.98 KB

Here I am keeping minimum 55 days data and after every 5 days , deleting oldest index

5

That looks ok.

Yes. But the data obviously won't move.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.