OK thank you very much. I will see this.
Hi again @antonopo
I have discovered the reason why your logs from metricbeat are going to metricbeat-7.2.0 policy. This is because elastic create a template for this kind of logs automatically, what you have to do is delete this template, you can do this with the following command:
DELETE _template/metricbeat-7.2.0
If this work for you please choose one reply as the solution of this topic to help another people who need it.
Thanks for your reply. I have the feeling that you are right.
The metricbeat-7.2.0 policy was deleted. Let's see until tomorrow if the new metricbeat indexed go automatically to my default metricbeat_policy.
Even if a new metricbeat will create it goes again to metricbeat-7.2.0 policy
You can check that the new metricbeat index that was created metricbeat-7.3.2-2019.09.13 has again as default policy the metricbeat-7.2.0
In every new metricbeat index that is creating is creating by default this template
metricbeat-7.2.0
Has been created again the metricbeat-7.2.0 template?
Yes in every new metricbeat index
Somewhere else must be this setting stored.
Hi again @antonopo
I think that this setting is stored in the metricbeat.yml config file, if Im not wrong you have to set this parameter setup.ilm.enabled: false
https://www.elastic.co/guide/en/beats/metricbeat/current/ilm.html
Yes you are right on this, i have set it to false.
setup.ilm.enabled: false
Do i have to set it to true?
You have to set is to false, if you set it as false, the indexes from Metricbeat shouldnt be added to any ilm.
###################### Metricbeat Configuration Example #######################
# This file is an example configuration file highlighting only the most common
# options. The metricbeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/metricbeat/index.html
#========================== Modules configuration ============================
metricbeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# Period on which files under path should be checked for changes
#reload.period: 10s
#==================== Elasticsearch template setting ==========================
setup.template.settings:
index.number_of_shards: 1
index.codec: best_compression
#_source.enabled: false
#================================ General =====================================
# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:
# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]
# Optional fields that you can specify to add additional information to the
# output.
#fields:
# env: staging
#============================== Dashboards =====================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: false
# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:
#============================== Kibana =====================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
# Kibana Host
host: "atdevxhv03.emea.nsn-net.net:5601"
# Scheme and port can be left out and will be set to the default (http and 5601)
# In case you specify and additional path, the scheme is required: http://localhost:5601/path
# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
#host: "localhost:5601"
# Kibana Space ID
# ID of the Kibana Space into which the dashboards should be loaded. By default,
# the Default Space will be used.
#space.id:
#============================= Elastic Cloud ==================================
# These settings simplify using Metricbeat with the Elastic Cloud (https://cloud.elastic.co/).
# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:
# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:
#================================ Outputs =====================================
# Configure what output to use when sending the data collected by the beat.
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
#hosts: ["localhost:9200"]
hosts: ["atdevxhv03.emea.nsn-net.net:9200", "it017060.it.alcatel-lucent.com:9200", "frelkxh01.fr.alcatel-lucent.com:9200", "atkvmxh01.emea.nsn-net.n
et:9200", "frkvmxh01.fr.alcatel-lucent.com:9200", "atdevxhv04.nsn-intra.net:9200", "itkvmxh01.emea.nsn-net.net:9200"]
# Optional protocol and basic auth credentials.
#protocol: "https"
#username: "elastic"
#password: "changeme"
#----------------------------- Logstash output --------------------------------
#output.logstash:
# The Logstash hosts
#hosts: ["localhost:5044"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"
# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"
#================================ Processors =====================================
# Configure processors to enhance or manipulate events generated by the beat.
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
#================================ Logging =====================================
# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug
# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]
#============================== Xpack Monitoring ===============================
# metricbeat can export internal metrics to a central Elasticsearch monitoring
# cluster. This requires xpack monitoring to be enabled in Elasticsearch. The
# reporting is disabled by default.
# Set to true to enable the monitoring reporter.
#monitoring.enabled: false
# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring.elasticsearch:
#================================= Migration ==================================
# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true
setup.ilm.enabled: falseThis is my metricbeat configuration
Try adding setup.template.enabled: false
After this maybe you would have to make a filter in logstash to create a new index each day automatically
Hi again,
I have already this on my logstash configuration
[root@atdevxhv03 ~]# cat /etc/logstash/conf.d/30-elasticsearch-output.conf
output {
elasticsearch {
hosts => ["atdevxhv03.emea.nsn-net.net:9200"]
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
Metricbeat has as output the elasticsearch and not the logstash. Is this gonna work if i will do the change as you said having as output the elasticsearch?
Yes, should work anyway but maybe elastcisearch just create one index for all your logs from Metricbeat. Im not sure but you can try
Hi again,
I tried this to heartbeat since i have the same problem like metricbeat. So no any new template were created but again is creating every time a new lifecycle policy
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.