Correct way to setup the beats index patterns?

Elastic Stack Beats
1

Hello and thanks in advance for the guidance. I've been trying to figure this out on my own for a while, but I'm not having any luck so I thought I'd just finally ask. What is the CORRECT way to setup the indexes for the beats in elasticsearch? I have 5-7 hosts in my lab (mix of windows and ubuntu) and I can deploy the beats to them just fine, but if I ever delete the metricbeat-7.10.1-2020.12.11-000001 index through Kibana elasticsearch then creates an index named metricbeat-7.10.1 which then breaks all of the sample dashboards.

I've read about "Loading the index template" at the following link -- > https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-template.html

Which led me to add the setup.template.enabled: true directive to my metricbeat.yml but it didn't seem to help.. I also tried false in case I was misunderstanding.. but that also did not help.

What DOES work, is stopping all instances of metribeat running on all devices in my environment.. then letting elastic create the correct index, then restarting all the instances. But I can't see that approach working at scale, which is what I'm trying to learn.

Here's my metricbeat.yml

metricbeat.config.modules:

  path: ${path.config}/modules.d/*.yml

  reload.enabled: true

output.elasticsearch:
  hosts: ["10.0.0.5:9200"]

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
2

HI,
Sorry, but I couldn't understand your goal here. Do you want to safely change the index name from metricbeat-7.10.1-2020.12.11-000001 to metricbeat-7.10.1 or is it something else?

3

Hey borna_talebi,

Thanks for the response.. no, when I start up the first beat in my lab (for instance metricbeat) I get the "correct" name of the index.. which is in the form of metricbeat-7.10.1-2020.12.11-000001 and all the default dashboards work perfectly. However, the second I manually delete the index metricbeat-7.10.1-2020.12.11-000001 to clear the data from within Kibana.. Elasticsearch somehow re-creates the new index in the shorter form of metricbeat-7.10.1 and the dashboards no longer work. I want to know where I need to configure (either beat or elasticsearch) to properly create the new index in the appropriate form of metricbeat-7.10.1-2020.12.11-000001 again so that the dashboards will work afterwards.

4

After deleting an index, try the below command and see if it creates the correct index for you:
metricbeat setup -c PATH_TO_.YML_CONFIG_FILE

Although dashboards usually work with index patterns like metricbeats-* so both of your index should work fine.
Could you check what index pattern your dashboards are using?

5

yeah.. they all use metricbeat-*, but.. when the metricbeat-7.10.1 index is used.. instead of the metricbeat-7.10.1-2020.12.11-000001 format.. I get failed shard messages on all the dashboards

6

Sorry for the late reply.

What's the full error message? can you share it here?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.