How to load index templates and dashboards from single place in beats 6.x

[vb4t]

Hello.

As elastic stack 6.0 will reach its GA soon I would like to know the best way to automatically load index templates and dashboards for all the beats from the single server.

Beats 6.x changed the way index templates and dasboards are loaded. Templates and dasboards are not distributed as separated packages anymore but "setup framework" is used instead. OK, but I can not load filebeat templates from filebeat, winlogbeat templates from winlogbeat ... I will have one linux deployment machine which should load all the templates and dashboards automatically with ansible.

As far I know there is one working "cleanest" solution now - build own lightway beats (it is actually very simple), name it for example setupbeat and use it to load fields.yml and dashboards from all other beats.

Is there a better way to do it? When will be elasticsearch-ansible for 6.0 available? Will it somehow support auto template and dashboard loading as it can do for 5.0?

Thanks

[tudor (Tudor Golubenco)]

Hi,

Sorry for the late answer. I have a couple of suggestions:

• It is possible to use any of the Beats with the setup command to load the dashboards for any other Beat, or even for all of them, like this:

./metricbeat setup -e  -E "setup.dashboards.url=https://staging.elastic.co/6.0.0-rc1-5306e42e/downloads/beats/beats-dashboards/beats-dashboards-6.0.0-rc1.zip" -E 'setup.dashboards.beat=""'

That makes metricbeat setup mostly equivalent to the old import_dasboards.

• For things like ES templates or ES pipeline definitions, it's not bad to let them be loaded automatically by the Beats, even when you have thousands of Beats. That's because each Beat checks if the template/pipeline is loaded before doing the PUT. The GET to check that the template/pipeline exist is cheap and shouldn't cause issues in ES.

• That said, since you probably have ansible roles for the Beats, it shouldn't be a big issue to just call them with an extra time with setup after each upgrade, perhaps from a dedicated machine like you said.

Let me know if this helps.

[vb4t]

Hello,

the "new design" of ES template importing is great for simple use cases like "beats -> elasticsearch cluster". But in larger deployments beats do not see the elasticsearch cluster. Typical: beats -> logstash -> kafka -> logstash -> elasticsearch cluster. Beats and even first logstash can be in different network zone, there are typically some security rules between DMZ routing ... and it is not wanted to let beats directly see the elasticsearch cluster. So in practice there will be following situations:

• Filebeat - no problem here, we have a large number of very different log files so we use our customized templates here
• Winlogbeat - installed on Windows machines, no direct access to ES, so can not load templates by itself, it is also not a linux app, so not possible to load templates from dedicated ansible machine either
• Metricbeat, ... otherbeats - can not load templates by itself but can be loaded from a dedicated ansible machine

I know there are reasons for auto generating ES templates but I think there should be something like template loader which could be used from one dedicated machine to load templates for all beats.

To sum up, currently to manually loading templates for 6.x I need to download all beats, extract fields.yml and use an arbitrary beat to import them? (Is there some URL for downloading fields.yml directly?)

Thanks.