# Having strange problems with "term lookup" filter **URL:** \ **Category:** Elasticsearch\ **Created:** [June 24, 2013, 6:16pm UTC](https://discuss.elastic.co/t/having-strange-problems-with-term-lookup-filter/12528 "2013-06-24T18:16:22Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![daveey](https://avatars.discourse-cdn.com/v4/letter/d/4da419/32.png) [@daveey](https://discuss.elastic.co/u/daveey)\ **Post date:** [June 24, 2013, 6:16pm UTC](https://discuss.elastic.co/t/having-strange-problems-with-term-lookup-filter/12528/1 "2013-06-24T18:16:22Z")
I am really confused, and keep getting erratic results when I use the term lookup filter. I am hoping someone could give me a model for reasoning about it, as I've been unable to find anything in the documentation that explains the behavior. I run a 5 node cluster, with a 50 shard index, replicated 2x. Notable elasticsearch.yml settings are: indices.cache.filter.terms.expire\_after\_write: 1s indices.cache.filter.terms.expire\_after\_access: 30s index.number\_of\_shards: 50 index.number\_of\_replicas: 1 I then have a document: POST: luna\_2/Domain/4?routing=15793206719&refresh=true {"public\_index\_ids":[370846021070]} The following query returns results: POST: luna\_2/Task/\_search?routing=15793206719 { "filter": { "terms": { "\_cache\_key": "foo", "\_cache": false, "index\_ids": [ 370846021070 ] } } } While this: { "filter": { "terms": { "\_cache\_key": "bar", "\_cache": false, "index\_ids": { "index": "luna\_2", "type": "Domain", "id": "4", "path": "public\_index\_ids" } } } } Behaves erratically. When I first built the index, it returned the correct results. I then modified luna\_2/Domain/4 to have an empty public\_index\_ids array, and re-issued the query. It continued to return the same results (even though it should have started returning none). I changed the cache\_key (even though I asked for the filter to not be cached) and it returned an empty result set. I then changed luna\_2/Domain/4 to the original value {"public\_index\_ids":[370846021070]} And now, the query is still returning an empty result set, even with a new cache key. I can't figure out what could be happening. Are there levels of caching that I am missing? It seems like the cache for the Domain/4 document should expire after 30 seconds, yet something is getting kept around. Are there any tools to inspect the cache, or issue a query that doesn't use the cache? Any help would be much appreciated -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [June 24, 2013, 8:32pm UTC](https://discuss.elastic.co/t/having-strange-problems-with-term-lookup-filter/12528/2 "2013-06-24T20:32:58Z")
Hey, can you create a gist, in order to reproduce the issue? Also, this might help (to explain at least some parts of that erratic behaviour): [Lookup Terms Filter \_cache parameter not being taken into account · Issue #3219 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/3219) --Alex On Mon, Jun 24, 2013 at 8:16 PM, daveey [daveey@gmail.com](mailto:daveey@gmail.com) wrote: > I am really confused, and keep getting erratic results when I use the term > lookup filter. I am hoping someone could give me a model for reasoning > about it, as I've been unable to find anything in the documentation that > explains the behavior. > > I run a 5 node cluster, with a 50 shard index, replicated 2x. > > Notable elasticsearch.yml settings are: > > indices.cache.filter.terms.expire\_after\_write: 1s > indices.cache.filter.terms.expire\_after\_access: 30s > index.number\_of\_shards: 50 > index.number\_of\_replicas: 1 > > I then have a document: > > POST: luna\_2/Domain/4?routing=15793206719&refresh=true > > {"public\_index\_ids":[370846021070]} > > The following query returns results: > > POST: luna\_2/Task/\_search?routing=15793206719 > { > "filter": { > "terms": { > "\_cache\_key": "foo", > "\_cache": false, > "index\_ids": [ > 370846021070 > ] > } > } > } > > While this: > > { > "filter": { > "terms": { > "\_cache\_key": "bar", > "\_cache": false, > "index\_ids": { > "index": "luna\_2", > "type": "Domain", > "id": "4", > "path": "public\_index\_ids" > } > } > } > } > > Behaves erratically. When I first built the index, it returned the correct > results. > > I then modified luna\_2/Domain/4 to have an empty public\_index\_ids array, > and re-issued the query. > > It continued to return the same results (even though it should have > started returning none). I changed the cache\_key (even though I asked for > the filter to not be cached) and it returned an empty result set. > > I then changed luna\_2/Domain/4 to the original value > {"public\_index\_ids":[370846021070]} > > And now, the query is still returning an empty result set, even with a new > cache key. > > I can't figure out what could be happening. Are there levels of caching > that I am missing? It seems like the cache for the Domain/4 document should > expire after 30 seconds, yet something is getting kept around. > > Are there any tools to inspect the cache, or issue a query that doesn't > use the cache? > > Any help would be much appreciated > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![daveey](https://avatars.discourse-cdn.com/v4/letter/d/4da419/32.png) [@daveey](https://discuss.elastic.co/u/daveey)\ **Post date:** [June 24, 2013, 9:19pm UTC](https://discuss.elastic.co/t/having-strange-problems-with-term-lookup-filter/12528/3 "2013-06-24T21:19:08Z")
Yes, this issue does explain a ton! I was assuming that \_cache: false would work, or that changing the looked up document would automatically invalidate the cached filter. I will set things up to manually invalidate the cache\_key, and see how that goes. Thanks very much! On Monday, June 24, 2013 1:32:58 PM UTC-7, Alexander Reelsen wrote: > Hey, > > can you create a gist, in order to reproduce the issue? > Also, this might help (to explain at least some parts of that erratic > behaviour): [Lookup Terms Filter \_cache parameter not being taken into account · Issue #3219 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/3219) > > --Alex > > On Mon, Jun 24, 2013 at 8:16 PM, daveey \<[dav...@gmail.com](mailto:dav...@gmail.com) \\>wrote: > > > I am really confused, and keep getting erratic results when I use the > > term lookup filter. I am hoping someone could give me a model for reasoning > > about it, as I've been unable to find anything in the documentation that > > explains the behavior. > > > > I run a 5 node cluster, with a 50 shard index, replicated 2x. > > > > Notable elasticsearch.yml settings are: > > > > indices.cache.filter.terms.expire\_after\_write: 1s > > indices.cache.filter.terms.expire\_after\_access: 30s > > index.number\_of\_shards: 50 > > index.number\_of\_replicas: 1 > > > > I then have a document: > > > > POST: luna\_2/Domain/4?routing=15793206719&refresh=true > > > > {"public\_index\_ids":[370846021070]} > > > > The following query returns results: > > > > POST: luna\_2/Task/\_search?routing=15793206719 > > { > > "filter": { > > "terms": { > > "\_cache\_key": "foo", > > "\_cache": false, > > "index\_ids": [ > > 370846021070 > > ] > > } > > } > > } > > > > While this: > > > > { > > "filter": { > > "terms": { > > "\_cache\_key": "bar", > > "\_cache": false, > > "index\_ids": { > > "index": "luna\_2", > > "type": "Domain", > > "id": "4", > > "path": "public\_index\_ids" > > } > > } > > } > > } > > > > Behaves erratically. When I first built the index, it returned the > > correct results. > > > > I then modified luna\_2/Domain/4 to have an empty public\_index\_ids array, > > and re-issued the query. > > > > It continued to return the same results (even though it should have > > started returning none). I changed the cache\_key (even though I asked for > > the filter to not be cached) and it returned an empty result set. > > > > I then changed luna\_2/Domain/4 to the original value > > {"public\_index\_ids":[370846021070]} > > > > And now, the query is still returning an empty result set, even with a > > new cache key. > > > > I can't figure out what could be happening. Are there levels of caching > > that I am missing? It seems like the cache for the Domain/4 document should > > expire after 30 seconds, yet something is getting kept around. > > > > Are there any tools to inspect the cache, or issue a query that doesn't > > use the cache? > > > > Any help would be much appreciated > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![daveey](https://avatars.discourse-cdn.com/v4/letter/d/4da419/32.png) [@daveey](https://discuss.elastic.co/u/daveey)\ **Post date:** [June 24, 2013, 10:20pm UTC](https://discuss.elastic.co/t/having-strange-problems-with-term-lookup-filter/12528/4 "2013-06-24T22:20:42Z")
Ok, I can now repro one of the problems, here is the gist: [Problem with term lookup filter · GitHub](https://gist.github.com/anonymous/5854123) I think the issue might be with the lookup ignoring the "routing" parameter. If I create the domain without a routing, things work fine. On Monday, June 24, 2013 2:19:08 PM UTC-7, daveey wrote: > Yes, this issue does explain a ton! I was assuming that \_cache: false > would work, or that changing the looked up document would automatically > invalidate the cached filter. I will set things up to manually invalidate > the cache\_key, and see how that goes. > > Thanks very much! > > On Monday, June 24, 2013 1:32:58 PM UTC-7, Alexander Reelsen wrote: > > > Hey, > > > > can you create a gist, in order to reproduce the issue? > > Also, this might help (to explain at least some parts of that erratic > > behaviour): [Lookup Terms Filter \_cache parameter not being taken into account · Issue #3219 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/3219) > > > > --Alex > > > > On Mon, Jun 24, 2013 at 8:16 PM, daveey [dav...@gmail.com](mailto:dav...@gmail.com) wrote: > > > > > I am really confused, and keep getting erratic results when I use the > > > term lookup filter. I am hoping someone could give me a model for reasoning > > > about it, as I've been unable to find anything in the documentation that > > > explains the behavior. > > > > > > I run a 5 node cluster, with a 50 shard index, replicated 2x. > > > > > > Notable elasticsearch.yml settings are: > > > > > > indices.cache.filter.terms.expire\_after\_write: 1s > > > indices.cache.filter.terms.expire\_after\_access: 30s > > > index.number\_of\_shards: 50 > > > index.number\_of\_replicas: 1 > > > > > > I then have a document: > > > > > > POST: luna\_2/Domain/4?routing=15793206719&refresh=true > > > > > > {"public\_index\_ids":[370846021070]} > > > > > > The following query returns results: > > > > > > POST: luna\_2/Task/\_search?routing=15793206719 > > > { > > > "filter": { > > > "terms": { > > > "\_cache\_key": "foo", > > > "\_cache": false, > > > "index\_ids": [ > > > 370846021070 > > > ] > > > } > > > } > > > } > > > > > > While this: > > > > > > { > > > "filter": { > > > "terms": { > > > "\_cache\_key": "bar", > > > "\_cache": false, > > > "index\_ids": { > > > "index": "luna\_2", > > > "type": "Domain", > > > "id": "4", > > > "path": "public\_index\_ids" > > > } > > > } > > > } > > > } > > > > > > Behaves erratically. When I first built the index, it returned the > > > correct results. > > > > > > I then modified luna\_2/Domain/4 to have an empty public\_index\_ids array, > > > and re-issued the query. > > > > > > It continued to return the same results (even though it should have > > > started returning none). I changed the cache\_key (even though I asked for > > > the filter to not be cached) and it returned an empty result set. > > > > > > I then changed luna\_2/Domain/4 to the original value > > > {"public\_index\_ids":[370846021070]} > > > > > > And now, the query is still returning an empty result set, even with a > > > new cache key. > > > > > > I can't figure out what could be happening. Are there levels of caching > > > that I am missing? It seems like the cache for the Domain/4 document should > > > expire after 30 seconds, yet something is getting kept around. > > > > > > Are there any tools to inspect the cache, or issue a query that doesn't > > > use the cache? > > > > > > Any help would be much appreciated > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:29am UTC](https://discuss.elastic.co/t/having-strange-problems-with-term-lookup-filter/12528/5 "2017-07-06T02:29:41Z")