Metricbeat behind logstash (index and dashboard generation)

[Trolldemorted]

Hi,

I have several servers with beats which talk to my logstash, and each should go into a separate index:

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}"
  }
}

As long as I run and finish metricbeat setup -e -E output.logstash.enabled=false -E output.elasticsearch.hosts=['localhost:9200'] -E setup.kibana.host=localhost:5601 before the first metricbeat sends data, it is all fun and games: It creates an index per day (e.g. metricbeat-7.6.2-2020.04.03-000001), creates the dashboards, and logstash uses that index to store the incoming events (why/how does that work, by the way?).

Unfortunately in my orchestration setup, I cannot easily run metricbeat setup before events arrive, so logstash has created an index called metricbeat-7.6.2 before it runs, and I cannot convince it to modify that index: it always says Exiting: resource 'metricbeat-7.6.2' exists, but it is not an alias.

Can I tell it to forcefully edit the index and make things work? I have tried hacks like "delete the index with curl and quickly run setup" (new data arrives too fast) and the options -E setup.template.overwrite=true and -E setup.ilm.overwrite=true, but it that did not help at all.

[spinscale (Alexander Reelsen)]

You need to set up the index template (the work done by metricbeat) before logstash starts indexing. You could also run a metricbeat setup run from the logstash host with the right configuration, as that one is able to reach elasticsearch - but you have to do this before the first indexing happens.

[Trolldemorted]

There is no way to do it after logstash has started? Isn't this a situation migrations always encounter? I could totally live with data loss or having to restart logstash once.