# Compound Indexes for User Based Data

**URL:** <https://discuss.elastic.co/t/compound-indexes-for-user-based-data/109887>\
**Category:** Elasticsearch\
**Created:** [December 1, 2017, 6:59am UTC](https://discuss.elastic.co/t/compound-indexes-for-user-based-data/109887 "2017-12-01T06:59:34Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Abhinav\_Tankha](https://avatars.discourse-cdn.com/v4/letter/a/ba9def/32.png) [@Abhinav\_Tankha](https://discuss.elastic.co/u/Abhinav_Tankha)\
**Post date:** [December 1, 2017, 6:59am UTC](https://discuss.elastic.co/t/compound-indexes-for-user-based-data/109887/1 "2017-12-01T06:59:35Z")

</div>

For user-based-data for which the search query will always have some user reference(eg. user-id) and additional query params is there a way to maintain compound indexes in elasticsearch to reduce the search space.

For our use case where there are millions of records for every user. We have created userId-mod based indexes to created buckets for users and added routing on user-id to restrict search queries to a single shard on elasticsearch for a user. Typical queries will always have user-id as a param followed with 2 or more filters.  
eg. user\_id = 12345, item\_purchase = 'Sony Headphones', item\_store = 'Chroma'

AFAIK, the segments for each - i.e user\_id, item\_purchase and item\_store will be searched independently based on a optimised order followed by a skip list merge.  
The search space for the fields(eg item\_purchase), which could be of low cardinality, would span across user-ids and would pollute the search space adding many more docIds than could possibly match.

Is there a way this can be handled better in elasticsearch. possible any options for compound indexes, where the key in the internal posting list has reference of the user-id as well.

I looked into the documentation and found a documentation touching upon this problem but couldnt get a solution for many users with many records and low cardinality fields -  
[https://www.elastic.co/guide/en/elasticsearch/guide/current/user-based.html](https://www.elastic.co/guide/en/elasticsearch/guide/current/user-based.html)

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [December 1, 2017, 7:39am UTC](https://discuss.elastic.co/t/compound-indexes-for-user-based-data/109887/2 "2017-12-01T07:39:26Z")

</div>

You should check out the lucene docs, as Lucene is already doing some optimizations around that (and more to come in future versions).

Also, check out index time sorting, see [https://www.elastic.co/guide/en/elasticsearch/reference/6.0/index-modules-index-sorting.html](https://www.elastic.co/guide/en/elasticsearch/reference/6.0/index-modules-index-sorting.html)

---

<div class="post-metadata">

**Author:** ![Abhinav\_Tankha](https://avatars.discourse-cdn.com/v4/letter/a/ba9def/32.png) [@Abhinav\_Tankha](https://discuss.elastic.co/u/Abhinav_Tankha)\
**Post date:** [December 4, 2017, 10:09am UTC](https://discuss.elastic.co/t/compound-indexes-for-user-based-data/109887/3 "2017-12-04T10:09:23Z")

</div>

Thanks @spinscale for sharing this info.

1. Is there any documentation on the effort on adding compound indexes in lucene and also its adoption in elasticsearch? Will keep a check on the progress.

2. The sorted index feature is in v6.0. What is the timeline to release the stable GA version?

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [December 4, 2017, 4:45pm UTC](https://discuss.elastic.co/t/compound-indexes-for-user-based-data/109887/4 "2017-12-04T16:45:01Z")

</div>

Just FYI: we keep this notice there mainly to give the feature some time to stabilize and especially to make sure that the API is what we need - which allows us to change the API more often.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 1, 2018, 4:45pm UTC](https://discuss.elastic.co/t/compound-indexes-for-user-based-data/109887/5 "2018-01-01T16:45:28Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.