Elastic not following their own advice to prevent sparsity?

Elastic Stack Beats
1

With the deprecation and eventual removal of "document types" from indices in ES 6 and 7 respectively, I am confused about filebeat's behaviour. For instance, if I am running filebeat to collect nginx and postgresql logs, those logs are still being submitted to the same filebeat index by default. ie. filebeat-{version}-{date}

To prevent sparsity in the indices, shouldn't the logs be written to separate indexes like so:
filebeat-{module/type}-{version}-{date}

To end up with indices like this:
filebeat-nginx-6.6.0-2019.03.04
filebeat-postgresql-6.6.0-2019.03.04

2

And as a follow-up, if the indexing is modified to split logs from different services into separate indices, does that have any impact on some of the newer features in Kibana like Logs or Infrastructure?

3

Quoting from my Twitter response:

it‘s a bit of a tradeoff: excess sparsity is an issue (but actually got better with newer lucene versions), but too many indices is probably worse. how many fields does your nginx and postgres index have — like 100? then I wouldn‘t worry

Does that make sense?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.