All modules log into the same index in metricbeat - best practise?

[owulff]

Hi there

I remember the following blog post about index vs type discussion:

Index vs. Type | Elastic

Especially one statement:

Do your documents have similar mappings? If no, use different indices.

IMHO, this applies to the different modules because only a few fields match with each other. Therefore, I'm wondering how does the current approach in metricbeat really scales in terms of memory/disk usage. Of course, I could update this within logstash and route it to different indices but then all dashboards are broken. What do you recommend?

Thanks
Oli

[ruflin]

You can already feed it now into different indices as the index name supports a format string. So you could use metricset.module for example as part of the index name. https://www.elastic.co/guide/en/beats/libbeat/current/config-file-format-type.html#_format_string_sprintf

In general, something we are actively discussing internally. Having one index per module can also lead to shard explosion.

About the dashboards: Having it under metricbeat-{module.name}-* indices the dashboard pattern could still be metricbeat-* and this should still work.

[owulff]

I know what you mean which shards explosion. At least it might make sense - if it's not already set - to have one shard per index.

Haven't yet migrated to 5.x so I hope it works.

Thanks
Oli

[ruflin]

You are probably interested in this discussion: https://github.com/elastic/beats/issues/3431