Elastic not following their own advice to prevent sparsity?

With the deprecation and eventual removal of "document types" from indices in ES 6 and 7 respectively, I am confused about filebeat's behaviour. For instance, if I am running filebeat to collect nginx and postgresql logs, those logs are still being submitted to the same filebeat index by default. ie. filebeat-{version}-{date}

To prevent sparsity in the indices, shouldn't the logs be written to separate indexes like so:

To end up with indices like this:

And as a follow-up, if the indexing is modified to split logs from different services into separate indices, does that have any impact on some of the newer features in Kibana like Logs or Infrastructure?

Quoting from my Twitter response:

it‘s a bit of a tradeoff: excess sparsity is an issue (but actually got better with newer lucene versions), but too many indices is probably worse. how many fields does your nginx and postgres index have — like 100? then I wouldn‘t worry

Does that make sense?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.