Index Name is not changing in metric beat

[sanju1323 (Sanjay Reddy)]

Hi,

I have installed metricbeat in my cluster and its working fine. Now I want to change the index name for the metricbeat.

For that I have changed index name in Outputs of metricbeat.reference.yml file as shown below

#================================ Outputs ======================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output -------------------------------
output.elasticsearch:
  # Boolean flag to enable or disable the output module.
  #enabled: true

  # Array of hosts to connect to.
  # Scheme and port can be left out and will be set to the default (http and 9200)
  # In case you specify and additional path, the scheme is required: http://localhost:9200/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:9200
  hosts: ["13.15.16.22:9200"]

  # Set gzip compression level.
  #compression_level: 0

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  username: "elastic"
  password: "changeme@dC"

  # Dictionary of HTTP parameters to pass within the url with index operations.
  #parameters:
    #param1: value1
    #param2: value2

  # Number of workers per Elasticsearch host.
  #worker: 1

  # Optional index name. The default is "metricbeat" plus date
  # and generates [metricbeat-]YYYY.MM.DD keys.
  # In case you modify this pattern you must update setup.template.name and setup.template.pattern accordingly.

  index: "metricbeat-%{[beat.version]}-instance1-%{+yyyy.MM.dd}"

  # Optional ingest node pipeline. By default no pipeline will be used.
  #pipeline: ""

  # Optional HTTP Path
  #path: "/elasticsearch"

  # Custom HTTP headers to add to each request
  #headers:
  #  X-My-Header: Contents of the header

  # Proxy server url
  #proxy_url: http://proxy:3128

  # The number of times a particular Elasticsearch index operation is attempted. If
  # the indexing operation doesn't succeed after this many retries, the events are
  # dropped. The default is 3.
  #max_retries: 3

  # The maximum number of events to bulk in a single Elasticsearch bulk API index request.
  # The default is 50.
  #bulk_max_size: 50

  # Configure http request timeout before failing an request to Elasticsearch.
  #timeout: 90

  # Use SSL settings for HTTPS. Default is true.
  #ssl.enabled: true

  # Configure SSL verification mode. If `none` is configured, all server hosts
  # and certificates will be accepted. In this mode, SSL based connections are
  # susceptible to man-in-the-middle attacks. Use only for testing. Default is
  # `full`.
  #ssl.verification_mode: full

  # List of supported/valid TLS versions. By default all TLS versions 1.0 up to
  # 1.2 are enabled.
  #ssl.supported_protocols: [TLSv1.0, TLSv1.1, TLSv1.2]

  # SSL configuration. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

  # Optional passphrase for decrypting the Certificate Key.
  #ssl.key_passphrase: ''

  # Configure cipher suites to be used for SSL connections
  #ssl.cipher_suites: []

  # Configure curve types for ECDHE based cipher suites
  #ssl.curve_types: []

But, when I start metricbeat using systemctl start metricbeat it is again indexing the data to the previous index which is index: "metricbeat-%{[beat.version]}-%{+yyyy.MM.dd}"

Please help me in changing the index name.

Regards,
Dharma Sanjay Reddy M.

[pierhugues (Pier-Hugues Pellerin)]

Hello,

At first, your config looks OK; I would check the following:

• Is the YAML alignment correct, a missing space could still return a valid config but would make the output ignore the index key.? You can verify what metricbeat sees in the configuration by running this command: ./metricbeat export config
• is Metricbeat reading the right configuration? Maybe try starting Metricbeat without systemd and use the -c to point the configuration file, also add `-e -to make it more verbose.

Thanks

[sanju1323 (Sanjay Reddy)]

Hi @pierhugues, thanks for the response,

By updating the templates in metricbeat.yml file, I'm able to index the data. But, not able to index all the fields due to mismatch in the metricbeat template file.
To update the metricbeat template file, I have tried changing the template name to metricbeat. I'm posting the lines that I have changed.

{
  "template": "metricbeat",
  "order": 1,
  "settings": {
    "index": {
      "codec": "best_compression",
      "mapping": {
        "total_fields": {
          "limit": "10000"
        }
      },
      "refresh_interval": "5s",
      "number_of_shards": "1"
    }
  },
  "mappings": {
    "metricbeat-*": {
      "date_detection": false,
      "dynamic_templates": [
        {.........

My Index name is metricbeat-instance1-%{+yyyy.MM.dd}
But still few fields are missing. Please let me know if I'm missing any changes in the template file.

Regards,
Dharma Sanjay Reddy M.