# Indexing strategy for sparse data **URL:** \ **Category:** Elasticsearch\ **Created:** [November 13, 2017, 8:53am UTC](https://discuss.elastic.co/t/indexing-strategy-for-sparse-data/107356 "2017-11-13T08:53:35Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![dandrestor](https://avatars.discourse-cdn.com/v4/letter/d/7cd45c/32.png) [@dandrestor](https://discuss.elastic.co/u/dandrestor)\ **Post date:** [November 13, 2017, 8:53am UTC](https://discuss.elastic.co/t/indexing-strategy-for-sparse-data/107356/1 "2017-11-13T08:53:35Z")
Hello, I have a design-related conundrum, and I'm hoping somebody might have already come across the same problem. We have set up ELK to ingest logs from a variety of sources. These come in different types (such as linux logs, nginx logs, etc) -- I will call these _groups_. Metrics will be parsed out of these logs. A minority of metrics are common and will be normalised across all groups, but most of them will be group-specific. We have as a requirement to have visualisations of these metrics. `norms` are not required on these fields, but I believe `doc_values` are (see below). The number of groups is expected to grow as the service gains acceptance in our enterprise. I'm not sure how to best set up our indexing strategy, though. I have the following alternatives: 1. Use a separate index series for each group (for example `log-linux-*`, `log-nginx-*`). This produces dense data in each index, however the number of shards/indexes on my hot nodes will scale with the number of groups -- which may become a problem in the future. 2. Use a common index for all data. This will produce a sparse data set but the number of shards will be minimal. I'm not sure how big of a problem this is, as far as I'm aware the only real problem is wasted disk space, but maybe you are aware of others? 3. Use a common index for all data, turning off `doc_values` for all fields except the normalised minority. I'm not sure if this will result in the metrics being unusable for visualisations. As far as I understand the `doc_values` are needed for all aggregations, so even an average would not work which would make a simple line graph over time impossible. Please correct me if I'm wrong. 4. Some other strategy I haven't thought of. I appreciate any thoughts or pointers you might have. //Dan ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 13, 2017, 9:04am UTC](https://discuss.elastic.co/t/indexing-strategy-for-sparse-data/107356/2 "2017-11-13T09:04:22Z")
> [@dandrestor](#): > > Use a separate index series for each group (for example log-linux-_, log-nginx-_). This produces dense data in each index, however the number of shards/indexes on my hot nodes will scale with the number of groups -- which may become a problem in the future. This depends on the data volume, but having lots of small indices and shards [is inefficient](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster). If you have a long retention period you can however get around this by switching to e.g. monthly indices. > [@dandrestor](#): > > Use a common index for all data. This will produce a sparse data set but the number of shards will be minimal. I'm not sure how big of a problem this is, as far as I'm aware the only real problem is wasted disk space, but maybe you are aware of others? This is a common approach, and what Beats do. While its can lead to increased size on disk in Elasticasearch 5.x, [this is being improved in the upcoming Elasticsearch 6.0 release](https://www.elastic.co/blog/minimize-index-storage-size-elasticsearch-6-0). As 6.0 is not far away, this is probably the approach I would recommend. ---
**Author:** ![dandrestor](https://avatars.discourse-cdn.com/v4/letter/d/7cd45c/32.png) [@dandrestor](https://discuss.elastic.co/u/dandrestor)\ **Post date:** [November 13, 2017, 10:03am UTC](https://discuss.elastic.co/t/indexing-strategy-for-sparse-data/107356/3 "2017-11-13T10:03:38Z")
Thanks Christian, that helps a lot. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 11, 2017, 10:03am UTC](https://discuss.elastic.co/t/indexing-strategy-for-sparse-data/107356/4 "2017-12-11T10:03:47Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.