Want to create monitoring indices for only 7 days

Elastic Observability Metrics
1

Hello,

I am monitoring my logstash instance through metricbeat monitoring.
I want to configure monitoring setting as such that only 7 days monitoring indices is created in my cluster in order to monitor for only 7 days and not more..

Also Is there a way that only 1 logstash monitoring index is created once a week or month?? I dont dont want daily creation of monitoring index.

2

Hi @Rakhshunda_Noorein_J,

I think you would want to have a look at setting up ILM to manage the index rollover. You would still be able to use logstash to push to the stream:

3

Hello @Rakhshunda_Noorein_J ,

Not sure which version of ELK you're using, 8.X vesrion bydefault creates monitoring indices for beats,logstash..etc.
I would like to know about your way to send data through metricbeat to monitor logstash and beats stats, alternatively can check in datastream.
This data streams monitoring indices are governed by default ILM policy, you can alter them as per your requirement under policy.

Index Management->Data stream->Monitoring indices.

image
image930×776 36.8 KB

4

I am using version 8.4.3, but my monitoring index creating is .monitoring-es-7-2023.09.22, .monitoring-kibana-7-2023.09.22, .monitoring-logstash-7-2023.09.22, .monitoring-beat-7-2023.09.22... They are using legacy index template(i.e. .monitoring-es, .monitoring-kibana, .monitoring-logstash) and not .monitoring-logstash-mb, .monitoring-es-mb, .monitoring-kibana-mb.

5

now I have seen both of the indices are generating,

image
image1063×229 19.8 KB

.ds-.monitoring-logstash-8-mb - this type of indices are hidden indices and

.monitoring-es-7-2023.09.22 - this type of indices are not hidden..

6

Ok, are you using xpack settings? , in your elasticsearch,kibana, beats and logsstash YML settings are you using xpack.monitoring.enabled: true setting?
Also do let know your yml settings and if your are using metricbeats to monitor then have you enable beats.yml,logstash.yml under modules. d folder? please share the settings used for monitoring.

7

Ok, nice.So its done, do share your solution for community reference.

8

my elasticsearch cluster and kibana cluster are self monitored, and logstash cluster is metricbeat monitored.

Yes I have enabled logstash.yml under modules. d folder. And It is monitoring properly.

logstash-xpack.yml

- module: logstash
  xpack.enabled: true
  period: 10s
  hosts: ["localhost:9600"]
  username: ""
  password: ""
9

my monitoring indices that is generating is for elasticsearch kibana and beat that is self monitored is as

image
image926×501 59.2 KB

where no ILM is attached

image

but for logstash indices generating is as

image
image887×388 26.7 KB

where ILM is attached.

image

**How do I Delete indices where ILM is not attached?? I have created an ILM for 3 days default policy, but it is giving ILM error as it is asking for an alias.**
10

This might help you ,here you can controll through templates, also for monitoring have you checked default reference files ex logsatsh.yml. There you will get more info. metricbeat.yml,logstash.yml settings need to look to see wht exactly you're doing for monitoring config. Above link would help you understand.

11

Hello,

To achieve your monitoring goals, you can configure Logstash's monitoring settings in your Elasticsearch cluster. To keep monitoring indices for only 7 days, set the index lifecycle management (ILM) policy accordingly. Additionally, you can use ILM to control the frequency of Logstash monitoring index creation, such as creating a new one weekly or monthly, ensuring that only one index is active at a time, rather than daily creations. This will help you manage your monitoring data efficiently.

1 Like
12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.