# Route query so that data for a shard is localized **URL:** \ **Category:** Elasticsearch\ **Created:** [August 3, 2014, 6:37pm UTC](https://discuss.elastic.co/t/route-query-so-that-data-for-a-shard-is-localized/19056 "2014-08-03T18:37:33Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![ElasticSearch\_Users\_](https://avatars.discourse-cdn.com/v4/letter/e/b4bc9f/32.png) [@ElasticSearch\_Users\_](https://discuss.elastic.co/u/ElasticSearch_Users_)\ **Post date:** [August 3, 2014, 6:37pm UTC](https://discuss.elastic.co/t/route-query-so-that-data-for-a-shard-is-localized/19056/1 "2014-08-03T18:37:33Z")
Hi, I have fairly large data and a ES cluster. Can I use some shard knowledge to execute queries so that only data relevant to a particular shard is fetched for that shard/node? I want to make sure that if I have a filter, then the values in the TermFilter only hold records that are relevant to the shard it will act upon. Is this a known problem? If so, how is it solved? Is there any performance implication in using the tree-like data mapping in ES? I am evaluating it now, and I wanted to know if it is feasible to maintain a treelike structure in ES, or just split it into multiple records or multiple indices? Thanks, Sandeep -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/aef299d8-65f2-4b34-a2ae-8c9abeb9a7b2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/aef299d8-65f2-4b34-a2ae-8c9abeb9a7b2%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [August 3, 2014, 7:17pm UTC](https://discuss.elastic.co/t/route-query-so-that-data-for-a-shard-is-localized/19056/2 "2014-08-03T19:17:05Z")
Have you consulted the docs > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. about the optimizations of term lookup for TermFilter? There are caches in use, and for term lookup, you can also use routing to select a particular shard. Regarding the "tree-like data mapping": ES rolls the tree notation into a flat format to make use of the Lucene API for fields in documents. There is no performance implication with this. If you decide to use an extraordinary high amount of fields (\>\>1000), you will notice each field consumes a bit of RAM, but this is not related to a "tree-like data mapping". Jörg On Sun, Aug 3, 2014 at 8:37 PM, 'Sandeep Ramesh Khanzode' via elasticsearch [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) wrote: > Hi, > > I have fairly large data and a ES cluster. Can I use some shard knowledge > to execute queries so that only data relevant to a particular shard is > fetched for that shard/node? I want to make sure that if I have a filter, > then the values in the TermFilter only hold records that are relevant to > the shard it will act upon. Is this a known problem? If so, how is it > solved? > > Is there any performance implication in using the tree-like data mapping > in ES? I am evaluating it now, and I wanted to know if it is feasible to > maintain a treelike structure in ES, or just split it into multiple records > or multiple indices? > > Thanks, > Sandeep > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/aef299d8-65f2-4b34-a2ae-8c9abeb9a7b2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/aef299d8-65f2-4b34-a2ae-8c9abeb9a7b2%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/aef299d8-65f2-4b34-a2ae-8c9abeb9a7b2%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/aef299d8-65f2-4b34-a2ae-8c9abeb9a7b2%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFy48Ga63bH3Q8bmOwa-sRH4yVVODOw9NhxJ0YQD8AC7A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFy48Ga63bH3Q8bmOwa-sRH4yVVODOw9NhxJ0YQD8AC7A%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![ElasticSearch\_Users\_](https://avatars.discourse-cdn.com/v4/letter/e/b4bc9f/32.png) [@ElasticSearch\_Users\_](https://discuss.elastic.co/u/ElasticSearch_Users_)\ **Post date:** [August 6, 2014, 6:34am UTC](https://discuss.elastic.co/t/route-query-so-that-data-for-a-shard-is-localized/19056/3 "2014-08-06T06:34:11Z")
Hi Jörg, Thanks, really appreciate the response and the link. I will do a small PoC with the approach given therein. Since we are pulling data from an index, I am assuming we will be limited the first time by disk speed. In the cache, if the data for the field that is cached has some updates (like a new value being added in the multi-valued field or removed), will the purge and re-cache automatically happen? I also think that I will need to enable the \_source field for updates to work? Is there any value to be had by making the columns to be doc\_values in this case? I read that doc\_values cannot be used for filtering purposes though. Please confirm. Please let me know your comments. Thanks again, Thanks, Sandeep On Monday, 4 August 2014 00:47:12 UTC+5:30, Jörg Prante wrote: > Have you consulted the docs > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-terms-filter.html#_terms_lookup_mechanism) > > about the optimizations of term lookup for TermFilter? > > There are caches in use, and for term lookup, you can also use routing to > select a particular shard. > > Regarding the "tree-like data mapping": ES rolls the tree notation into a > flat format to make use of the Lucene API for fields in documents. There is > no performance implication with this. If you decide to use an extraordinary > high amount of fields (\>\>1000), you will notice each field consumes a bit > of RAM, but this is not related to a "tree-like data mapping". > > Jörg > > On Sun, Aug 3, 2014 at 8:37 PM, 'Sandeep Ramesh Khanzode' via > elasticsearch \<[elasti...@googlegroups.com](mailto:elasti...@googlegroups.com) \\> wrote: > > > Hi, > > > > I have fairly large data and a ES cluster. Can I use some shard knowledge > > to execute queries so that only data relevant to a particular shard is > > fetched for that shard/node? I want to make sure that if I have a filter, > > then the values in the TermFilter only hold records that are relevant to > > the shard it will act upon. Is this a known problem? If so, how is it > > solved? > > > > Is there any performance implication in using the tree-like data mapping > > in ES? I am evaluating it now, and I wanted to know if it is feasible to > > maintain a treelike structure in ES, or just split it into multiple records > > or multiple indices? > > > > Thanks, > > Sandeep > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/aef299d8-65f2-4b34-a2ae-8c9abeb9a7b2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/aef299d8-65f2-4b34-a2ae-8c9abeb9a7b2%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/aef299d8-65f2-4b34-a2ae-8c9abeb9a7b2%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/aef299d8-65f2-4b34-a2ae-8c9abeb9a7b2%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1b2c3fb0-a770-4857-9729-1da34a9baf04%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1b2c3fb0-a770-4857-9729-1da34a9baf04%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:10am UTC](https://discuss.elastic.co/t/route-query-so-that-data-for-a-shard-is-localized/19056/4 "2017-07-06T01:10:36Z")