# Elastic not following their own advice to prevent sparsity?

**URL:** <https://discuss.elastic.co/t/elastic-not-following-their-own-advice-to-prevent-sparsity/170760>\
**Category:** Beats\
**Created:** [March 4, 2019, 4:05pm UTC](https://discuss.elastic.co/t/elastic-not-following-their-own-advice-to-prevent-sparsity/170760 "2019-03-04T16:05:49Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![agx](https://avatars.discourse-cdn.com/v4/letter/a/e19adc/32.png) [@agx](https://discuss.elastic.co/u/agx)\
**Post date:** [March 4, 2019, 4:05pm UTC](https://discuss.elastic.co/t/elastic-not-following-their-own-advice-to-prevent-sparsity/170760/1 "2019-03-04T16:05:49Z")

</div>

With the deprecation and eventual removal of "document types" from indices in ES 6 and 7 respectively, I am confused about filebeat's behaviour. For instance, if I am running filebeat to collect nginx and postgresql logs, those logs are still being submitted to the same filebeat index by default. ie. filebeat-{version}-{date}

To prevent sparsity in the indices, shouldn't the logs be written to separate indexes like so:  
filebeat-{module/type}-{version}-{date}

To end up with indices like this:  
filebeat-nginx-6.6.0-2019.03.04  
filebeat-postgresql-6.6.0-2019.03.04

---

<div class="post-metadata">

**Author:** ![agx](https://avatars.discourse-cdn.com/v4/letter/a/e19adc/32.png) [@agx](https://discuss.elastic.co/u/agx)\
**Post date:** [March 4, 2019, 5:18pm UTC](https://discuss.elastic.co/t/elastic-not-following-their-own-advice-to-prevent-sparsity/170760/2 "2019-03-04T17:18:29Z")

</div>

And as a follow-up, if the indexing is modified to split logs from different services into separate indices, does that have any impact on some of the newer features in Kibana like Logs or Infrastructure?

---

<div class="post-metadata">

**Author:** ![xeraa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xeraa/32/48181_2.png) [@xeraa](https://discuss.elastic.co/u/xeraa)\
**Post date:** [March 8, 2019, 7:09pm UTC](https://discuss.elastic.co/t/elastic-not-following-their-own-advice-to-prevent-sparsity/170760/3 "2019-03-08T19:09:08Z")

</div>

Quoting from [my Twitter response](https://twitter.com/xeraa/status/1102738914555629568):

> it‘s a bit of a tradeoff: excess sparsity is an issue (but actually got better with newer lucene versions), but too many indices is probably worse. how many fields does your nginx and postgres index have — like 100? then I wouldn‘t worry

Does that make sense?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 5, 2019, 9:09pm UTC](https://discuss.elastic.co/t/elastic-not-following-their-own-advice-to-prevent-sparsity/170760/4 "2019-04-05T21:09:08Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.