Hi Team,
I have metricbeat installed on my servers to send data to ES. I want to have daily indexes for each day. Along with this, I also want to delete the indexes after few days using ILM. I am attaching my metricbeat.yml configuration.
Note: I have 3 ES nodes as cluster and 2 nodes for Logstash and Kibana.
> ###################### Metricbeat Configuration Example #######################
>
> # This file is an example configuration file highlighting only the most common
> # options. The metricbeat.reference.yml file from the same directory contains all the
> # supported options with more comments. You can use it as a reference.
> #
> # You can find the full configuration reference here:
> # https://www.elastic.co/guide/en/beats/metricbeat/index.html
>
> # =========================== Modules configuration ============================
>
> metricbeat.config.modules:
> # Glob pattern for configuration loading
> path: ${path.config}/modules.d/*.yml
>
> # Set to true to enable config reloading
> reload.enabled: false
>
> # Period on which files under path should be checked for changes
> #reload.period: 10s
>
> # ======================= Elasticsearch template setting =======================
> setup.template.name: "metric"
> setup.template.fields: "fields.yml"
> setup.template.overwrite: true
> setup.template.pattern: "metricbeateoprodsec-*"
> setup.template.settings:
> index.number_of_shards: 3
> index.number_of_replicas: 1
> index.codec: best_compression
> #_source.enabled: false
>
>
> # ================================== General ===================================
>
> # The name of the shipper that publishes the network data. It can be used to group
> # all the transactions sent by a single shipper in the web interface.
> #name:
>
> # The tags of the shipper are included in their own field with each
> # transaction published.
> #tags: ["service-X", "web-tier"]
>
> # Optional fields that you can specify to add additional information to the
> # output.
> #fields:
> # env: staging
>
> # ================================= Dashboards =================================
> # These settings control loading the sample dashboards to the Kibana index. Loading
> # the dashboards is disabled by default and can be enabled either by setting the
> # options here or by using the `setup` command.
> #setup.dashboards.enabled: true
>
> # The URL from where to download the dashboards archive. By default this URL
> # has a value which is computed based on the Beat name and version. For released
> # versions, this URL points to the dashboard archive on the artifacts.elastic.co
> # website.
> #setup.dashboards.url:
>
> # =================================== Kibana ===================================
>
> # Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
> # This requires a Kibana endpoint configuration.
> setup.kibana:
>
> # Kibana Host
> # Scheme and port can be left out and will be set to the default (http and 5601)
> # In case you specify and additional path, the scheme is required: http://localhost:5601/path
> # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
> host: "http://cms-public-nlb-bae35f5967957d10.elb.us-east-1.amazonaws.com/:5601"
> username: "admin"
> password: "admin"
>
> # Kibana Space ID
> # ID of the Kibana Space into which the dashboards should be loaded. By default,
> # the Default Space will be used.
> #space.id:
>
> # =============================== Elastic Cloud ================================
>
> # These settings simplify using Metricbeat with the Elastic Cloud (https://cloud.elastic.co/).
>
> # The cloud.id setting overwrites the `output.elasticsearch.hosts` and
> # `setup.kibana.host` options.
> # You can find the `cloud.id` in the Elastic Cloud web UI.
> #cloud.id:
>
> # The cloud.auth setting overwrites the `output.elasticsearch.username` and
> # `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
> #cloud.auth:
>
> # ================================== Outputs ===================================
>
> # Configure what output to use when sending the data collected by the beat.
>
> # ---------------------------- Elasticsearch Output ----------------------------
> output.elasticsearch:
> # Array of hosts to connect to.
> hosts: ["https://cms-public-nlb-bae35f5967957d10.elb.us-east-1.amazonaws.com:9200"]
> indices:
> - index: "metricbeateoprodsec-%{[agent.version]}-%{+yyyy.MM.dd}"
> # Protocol - either `http` (default) or `https`.
> protocol: "https"
>
> # Authentication credentials - either API key or username/password.
> #api_key: "id:api_key"
> username: "admin"
> password: "admin"
> ssl.verification_mode: none
>
> # ------------------------------ Logstash Output -------------------------------
> #output.logstash:
> # The Logstash hosts
> #hosts: ["localhost:5044"]
>
> # Optional SSL. By default is off.
> # List of root certificates for HTTPS server verifications
> #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
>
> # Certificate for SSL client authentication
> #ssl.certificate: "/etc/pki/client/cert.pem"
>
> # Client Certificate Key
> #ssl.key: "/etc/pki/client/cert.key"
>
> # ================================= Processors =================================
>
> # Configure processors to enhance or manipulate events generated by the beat.
>
> processors:
> - add_host_metadata: ~
> - add_cloud_metadata: ~
> - add_docker_metadata: ~
> - add_kubernetes_metadata: ~
>
>
> # ================================== Logging ===================================
>
> # Sets log level. The default log level is info.
> # Available log levels are: error, warning, info, debug
> #logging.level: debug
>
> # At debug level, you can selectively enable logging only for some components.
> # To enable all selectors use ["*"]. Examples of other selectors are "beat",
> # "publisher", "service".
> #logging.selectors: ["*"]
>
> # ============================= X-Pack Monitoring ==============================
> # Metricbeat can export internal metrics to a central Elasticsearch monitoring
> # cluster. This requires xpack monitoring to be enabled in Elasticsearch. The
> # reporting is disabled by default.
>
> # Set to true to enable the monitoring reporter.
> #monitoring.enabled: false
>
> # Sets the UUID of the Elasticsearch cluster under which monitoring data for this
> # Metricbeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
> # is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
> #monitoring.cluster_uuid:
>
> # Uncomment to send the metrics to Elasticsearch. Most settings from the
> # Elasticsearch output are accepted here as well.
> # Note that the settings should point to your Elasticsearch *monitoring* cluster.
> # Any setting that is not set is automatically inherited from the Elasticsearch
> # output configuration, so if you have the Elasticsearch output configured such
> # that it is pointing to your Elasticsearch monitoring cluster, you can simply
> # uncomment the following line.
> #monitoring.elasticsearch:
>
> # ============================== Instrumentation ===============================
>
> # Instrumentation support for the metricbeat.
> #instrumentation:
> # Set to true to enable instrumentation of metricbeat.
> #enabled: false
>
> # Environment in which metricbeat is running on (eg: staging, production, etc.)
> #environment: ""
>
> # APM Server hosts to report instrumentation results to.
> #hosts:
> # - http://localhost:8200
>
> # API Key for the APM Server(s).
> # If api_key is set then secret_token will be ignored.
> #api_key:
>
> # Secret token for the APM Server(s).
> #secret_token:
>
>
> # ================================= Migration ==================================
>
> # This allows to enable 6.7 migration aliases
> #migration.6_to_7.enabled: true
Below this is for 8.x but should work for 7.17 as well...
And I always add that although renaming default indices is supported however if you are just getting to know The Elastic Stack there are tons of reasons to just use the defaults, when you move away from the defaults it is critical you understand all those relationship.
Another approach is simply tagging these host(s) with a tag tags: ["metricbeateoprodsec"]
and leave everything as default and just filter by the tag in each of the areas and set the metric ILM policy to what you want
These are all the things affected when changing index / data stream names
Data Stream
Index Template
ILM Policy
Dashboards
Infrastructure App.
Probably more I am not remembering
So assuming you want to do this...Leverage the framework
Here is my fully functional metricbeat.yml
You will want to run setup Once then turn off the overwrite and dashboards loading.
Run setup first filebeat setup -e
then start filebeat filebeat -e
or systemctl start filebeat
# =========================== Modules configuration ============================
metricbeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# Period on which files under path should be checked for changes
#reload.period: 10s
# ======================= Elasticsearch template setting =======================
setup.template.enabled: true
setup.template.name: "metricbeateoprodsec-%{[agent.version]}"
setup.template.pattern: "metricbeateoprodsec-%{[agent.version]}"
setup.template.overwrite: true ## <- Set this to true Once then take out or set to false.
setup.ilm.policy_name: metricbeateoprodsec-policy
setup.template.settings:
index.number_of_shards: 1
index.number_of_replicas: 1
index.codec: best_compression
#_source.enabled: false
# Important if you want the dashboards to work
# Set this to true Once then set to false. Or comment out BOTH lines, Otherwise metricbeat will reload the dashboards EVERY TIME!
setup.dashboards.enabled: true
setup.dashboards.index: "metricbeateoprodsec-*"
# =================================== Kibana ===================================
setup.kibana:
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["http://localhost:9200"]
index: "metricbeateoprodsec-%{[agent.version]}"
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
Now you have a Data Stream, Index Template and ILM Policy and have set the Built in dashboards to work with all the changes
Hi,
I have mentioned the same file as mentioned by you, however I am only getting index as metricbeat-7.17.6 and not the string eoprod combined with it.
Unless you show exactly what you did every single command, the command output and every configuration we can't help. Just saying it does not work does not provide enough information to help.
It does assume that you understand you need to run filebeat setup -e before starting filebeat.
The example I gave above works if you follow it very specifically.
Also, just to be clear, this is a community forum... It's not paid support so sometimes your questions may not be able to be answered.
Hi Stephen,
I have followed the same metricbeat.yml file as mentioned by you. It has created a ILM policy as well, however the name of the indexes are as below:
###################### Metricbeat Configuration Example #######################
# This file is an example configuration file highlighting only the most common
# options. The metricbeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/metricbeat/index.html
# =========================== Modules configuration ============================
metricbeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# Period on which files under path should be checked for changes
#reload.period: 10s
# ======================= Elasticsearch template setting =======================
#setup.template.settings:
#index.number_of_shards: 1
# index.codec: best_compression
#_source.enabled: false
setup.template.enabled: true
setup.template.name: "metricbeateoprod-%{[agent.version]}"
setup.template.pattern: "metricbeateoprod-%{[agent.version]}"
setup.template.overwrite: true ## <- Set this to true Once then take out or set to false.
setup.ilm.policy_name: metricbeateoprod-policy
setup.template.settings:
index.number_of_shards: 3
index.number_of_replicas: 1
index.codec: best_compression
#_source.enabled: false
# Important if you want the dashboards to work
# Set this to true Once then set to false. Or comment out BOTH lines, Otherwise metricbeat will reload the dashboards EVERY TIME!
setup.dashboards.enabled: true
setup.dashboards.index: "metricbeateoprod-*"
# ================================== General ===================================
# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:
# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]
# Optional fields that you can specify to add additional information to the
# output.
#fields:
# env: staging
# ================================= Dashboards =================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: false
# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:
# =================================== Kibana ===================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
# Kibana Host
# Scheme and port can be left out and will be set to the default (http and 5601)
# In case you specify and additional path, the scheme is required: http://localhost:5601/path
# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
host: "http://xxx.xxx.xxx.xxx:5601"
username: "admin"
password: "admin"
# Kibana Space ID
# ID of the Kibana Space into which the dashboards should be loaded. By default,
# the Default Space will be used.
#space.id:
# =============================== Elastic Cloud ================================
# These settings simplify using Metricbeat with the Elastic Cloud (https://cloud.elastic.co/).
# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:
# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:
# ================================== Outputs ===================================
# Configure what output to use when sending the data collected by the beat.
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["https://xxx.xxx.xxx.xxx:9200"]
username: "admin"
password: "admin"
index: "metricbeateoprod-%{[agent.version]}"
ssl.verification_mode: none
# Protocol - either `http` (default) or `https`.
protocol: "https"
# Authentication credentials - either API key or username/password.
#api_key: "id:api_key"
#username: "elastic"
#password: "changeme"
# ------------------------------ Logstash Output -------------------------------
#output.logstash:
# The Logstash hosts
#hosts: ["localhost:5044"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"
# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"
# ================================= Processors =================================
# Configure processors to enhance or manipulate events generated by the beat.
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
# ================================== Logging ===================================
# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug
# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publisher", "service".
#logging.selectors: ["*"]
# ============================= X-Pack Monitoring ==============================
# Metricbeat can export internal metrics to a central Elasticsearch monitoring
# cluster. This requires xpack monitoring to be enabled in Elasticsearch. The
# reporting is disabled by default.
# Set to true to enable the monitoring reporter.
#monitoring.enabled: false
# Sets the UUID of the Elasticsearch cluster under which monitoring data for this
# Metricbeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
#monitoring.cluster_uuid:
# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring.elasticsearch:
# ============================== Instrumentation ===============================
# Instrumentation support for the metricbeat.
#instrumentation:
# Set to true to enable instrumentation of metricbeat.
#enabled: false
# Environment in which metricbeat is running on (eg: staging, production, etc.)
#environment: ""
# APM Server hosts to report instrumentation results to.
#hosts:
# - http://localhost:8200
# API Key for the APM Server(s).
# If api_key is set then secret_token will be ignored.
#api_key:
# Secret token for the APM Server(s).
#secret_token:
# ================================= Migration ==================================
# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true
I am using ELK Stack 7.17.6 version.
After making changes to the yml file, i have restarted the metricbeat service after deleting the index template as well.
output.elasticsearch.index: "metricbeateoprod-%{[agent.version]}"
setup.template.name: "metricbeateoprod-%{[agent.version]}"
setup.template.pattern: "metricbeateoprod-%{[agent.version]}"
setup.template.overwrite: true ## <- Set this to true Once then take out or set to false.
setup.template.enabled: true ## <- Set this to true Once then take out or set to false.
setup.ilm.policy_name: "metricbeateoprod-%{[agent.version]}"
setup.ilm.overwrite: true ## <- Set this to true Once then take out or set to false.
setup.ilm.enabled: true ## <- Set this to true Once then take out or set to false.
setup.ilm.policy_name: "metricbeateoprod-%{[agent.version]}"
setup.ilm.rollover_alias: "metricbeateoprod-%{[agent.version]}" ## this will be used to rollover to new index name as date-000002
setup.ilm.pattern: "{now/d}-000001" ## this is setup with date-000001 as first index name
What I did is
setup metricbeat.yml like this only on A system for initial setup.
started metricbeat "systemctl start metricbeat"
stop metricbeat "systemctl stop metricbeat"
edited metricbeat.yml, disable few line as required.
now all other metricbeat do not need any such thing. they just need output.index name ""metricbeateoprod-%{[agent.version]}" which is now alias to index ""metricbeateoprod-8.5.2-date-0000001"
@elasticforme I can see what you did will work but without running setup actually the dashboards and other assets will not be properly configured. (Example out of the box dashboards would not work without setup)
So I still think the correct steps is to use your configuration and specifically run setup.
Then edit those specific settings and then properly start metricbeat.
oh ya. I do not use or load premade dashboard hence I do not use that setup steps.
yes I was so confuse when I did this for 8.5.2 as it created data stream and .ds-* index and I can't find out the way it was working before.
I was freaking out then suddenly show by clicking each button.
then read up about it and decided to test out this data stream part of it
@Ankita_Pachauri Apologies this took too long it should not be this hard confusing (and I made it worse!)
Thanks @elasticforme For helping and below is the official method for changing the index names and still support OOTB dashboards etc.
Here is fully tested on 7.17.3
First you must clean up all the metricbeat indices, index templates, ILM policies from previous tries.
Then you will use this config and run setup, You need to run setup it will take a while as it will reload all the dashboards. NOTE:setup only need to be run Once, not on every host, just 1 time for all hosts.
setup.ilm.enabled: true
setup.ilm.overwrite: true ## <-- After Setup set to false
setup.ilm.policy_name: metricbeateoprodsec-policy
setup.ilm.rollover_alias: "metricbeateoprodsec-%{[agent.version]}"
setup.ilm.pattern: "{now/d}-000001"
setup.template.enabled: true
setup.template.overwrite: true ## <-- After Setup set to false
setup.template.name: "metricbeateoprodsec-%{[agent.version]}"
setup.template.pattern: "metricbeateoprodsec-%{[agent.version]}-*"
setup.template.settings:
index.number_of_shards: 1
index.number_of_replicas: 1
index.codec: best_compression
setup.dashboards.enabled: true ## <-- After Setup set to false
setup.dashboards.index: "metricbeateoprodsec-*"
Using Config Above
metricbeat setup -e
Afterwards you should see the Index Template, The Index and the ILM Policy.
Just go into the UI and adjust the ILM policy to whatever you like.
ILM is available for all the beats out of the box.... Really, it would be my suggestion that you would just install them and use them with the default settings, learn how they all work and then adjust from there. If you use the defaults defaults and not worry about these individual index names You would probably make much more progress. You could just as easily put a tag with the source name and then filter on the dashboards by those tags.
Every beat has default ILM policies and you can just go into the UI and change those policies to whatever you like.
If you need details, you should open specific threads on each question on different topics.
Perhaps stepping back and learning more about the overall concepts would help. Perhaps take some of the free training and there's lots of webinars and blogs on index lifecycle management.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
