I am monitoring my logstash instance through metricbeat monitoring.
I want to configure monitoring setting as such that only 7 days monitoring indices is created in my cluster in order to monitor for only 7 days and not more..
Also Is there a way that only 1 logstash monitoring index is created once a week or month?? I dont dont want daily creation of monitoring index.
Not sure which version of ELK you're using, 8.X vesrion bydefault creates monitoring indices for beats,logstash..etc.
I would like to know about your way to send data through metricbeat to monitor logstash and beats stats, alternatively can check in datastream.
This data streams monitoring indices are governed by default ILM policy, you can alter them as per your requirement under policy.
Index Management->Data stream->Monitoring indices.
I am using version 8.4.3, but my monitoring index creating is .monitoring-es-7-2023.09.22, .monitoring-kibana-7-2023.09.22, .monitoring-logstash-7-2023.09.22, .monitoring-beat-7-2023.09.22... They are using legacy index template(i.e. .monitoring-es, .monitoring-kibana, .monitoring-logstash) and not .monitoring-logstash-mb, .monitoring-es-mb, .monitoring-kibana-mb.
Ok, are you using xpack settings? , in your elasticsearch,kibana, beats and logsstash YML settings are you using xpack.monitoring.enabled: true setting?
Also do let know your yml settings and if your are using metricbeats to monitor then have you enable beats.yml,logstash.yml under modules. d folder? please share the settings used for monitoring.
This might help you ,here you can controll through templates, also for monitoring have you checked default reference files ex logsatsh.yml. There you will get more info. metricbeat.yml,logstash.yml settings need to look to see wht exactly you're doing for monitoring config. Above link would help you understand.
To achieve your monitoring goals, you can configure Logstash's monitoring settings in your Elasticsearch cluster. To keep monitoring indices for only 7 days, set the index lifecycle management (ILM) policy accordingly. Additionally, you can use ILM to control the frequency of Logstash monitoring index creation, such as creating a new one weekly or monthly, ensuring that only one index is active at a time, rather than daily creations. This will help you manage your monitoring data efficiently.