# Facets with all\_terms return deleted terms **URL:** \ **Category:** Elasticsearch\ **Created:** [March 21, 2013, 3:43pm UTC](https://discuss.elastic.co/t/facets-with-all-terms-return-deleted-terms/11245 "2013-03-21T15:43:59Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![the\_mouette](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/the_mouette/32/2422_2.png) [@the\_mouette](https://discuss.elastic.co/u/the_mouette)\ **Post date:** [March 21, 2013, 3:43pm UTC](https://discuss.elastic.co/t/facets-with-all-terms-return-deleted-terms/11245/1 "2013-03-21T15:43:59Z")
Hi, I'm building a search/filter interface based on _elasticsearch 0.20.X_. I've got books with following schema : - title - author - publisher I sometimes need to remove all _books_ of given _publisher_ for various reasons. Another business requirement is that all available publishers should be listed under the publisher filter, even when other filter criteria excludes all books. I use _all\_terms_ facets and everything went fine until I removed a publisher : it is still listed in the facet result. Is there any way to reset facet list ? Julien. Some sample data: {author: "Terry Pratchet", title: "Discworld 1", publisher: "Harper Collins" } {author: "Terry Pratchet", title: "Discworld 2", publisher: "Harper Collins" } {author: "Terry Pratchet", title: "Discworld 3", publisher: "Harper Collins" } {author: "Terry Pratchet", title: "Discworld 4", publisher: "Harper Collins" } {author: "Douglas Adams", title: "H2G2", publisher: "Geoffrey Perkins"} The following request list all publisher, event the deleted ones. $ curl -XPOST '[http://localhost:9200/books/\_search](http://localhost:9200/books/_search)' -d '{ "from":0, "size":50, "query": { "match\_all":{} }, "facets": { "publisher": { "terms": { "field": "publisher", "size": 70, "all\_terms": true, "order": "term" } } }' -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [March 21, 2013, 3:47pm UTC](https://discuss.elastic.co/t/facets-with-all-terms-return-deleted-terms/11245/2 "2013-03-21T15:47:36Z")
How do you delete a publisher? What does the command look like? Do you optimize the index after deletion? Jörg Am 21.03.13 16:43, schrieb [the.mouette@gmail.com](mailto:the.mouette@gmail.com): > Hi, > > I'm building a search/filter interface based on /elasticsearch 0.20.X/. > > I've got books with following schema : > > - title > - author > - publisher > > I sometimes need to remove all _books_ of given _publisher_ for > various reasons. > Another business requirement is that all available publishers should > be listed under the publisher filter, even when other filter criteria > excludes all books. > > I use _all\_terms_ facets and everything went fine until I removed a > publisher : it is still listed in the facet result. > > Is there any way to reset facet list ? > > Julien. > > Some sample data: > > | > {author:"Terry Pratchet",title:"Discworld 1",publisher:"Harper Collins"} > {author:"Terry Pratchet",title:"Discworld 2",publisher:"Harper Collins"} > {author:"Terry Pratchet",title:"Discworld 3",publisher:"Harper Collins"} > {author:"Terry Pratchet",title:"Discworld 4",publisher:"Harper Collins"} > {author:"Douglas Adams",title:"H2G2",publisher:"Geoffrey Perkins"} > | > > The following request list all publisher, event the deleted ones. > > | > $ curl -XPOST '[http://localhost:9200/books/\_search'-d](http://localhost:9200/books/_search'-d) '{ > "from":0, > "size":50, > "query": { "match\_all":{} }, > "facets": { > "publisher": { > "terms": { > "field": "publisher", > "size": 70, > "all\_terms": true, > "order": "term" > } > } > }' > > | > > -- > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [March 21, 2013, 4:05pm UTC](https://discuss.elastic.co/t/facets-with-all-terms-return-deleted-terms/11245/3 "2013-03-21T16:05:45Z")
> I use all\_terms facets and everything went fine until I removed a > publisher : it is still listed in the facet result. The terms list includes terms in documents that have been deleted. Deleting a document just marks it as deleted and excludes it from future searches. Only once the segment that the document was in is rewritten will the doc (and the terms) be truly expunged. That happens automatically in the background as segments are merged, or you can force it with the optimize API (which can be quite heavy) clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![the\_mouette](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/the_mouette/32/2422_2.png) [@the\_mouette](https://discuss.elastic.co/u/the_mouette)\ **Post date:** [March 21, 2013, 4:10pm UTC](https://discuss.elastic.co/t/facets-with-all-terms-return-deleted-terms/11245/4 "2013-03-21T16:10:24Z")
Wahoo, that was fast ! I'm using Elastica for deletion, I'll dig a bit to find out the exact query. Thx Le jeudi 21 mars 2013 16:47:36 UTC+1, Jörg Prante a écrit : > How do you delete a publisher? What does the command look like? > > Do you optimize the index after deletion? > > J?rg > > Am 21.03.13 16:43, schrieb [the.m...@gmail.com](mailto:the.m...@gmail.com) \: > > > Hi, > > > > I'm building a search/filter interface based on /elasticsearch 0.20.X/. > > > > I've got books with following schema : > > > > - title > > - author > > - publisher > > > > I sometimes need to remove all _books_ of given _publisher_ for > > various reasons. > > Another business requirement is that all available publishers should > > be listed under the publisher filter, even when other filter criteria > > excludes all books. > > > > I use _all\_terms_ facets and everything went fine until I removed a > > publisher : it is still listed in the facet result. > > > > Is there any way to reset facet list ? > > > > Julien. > > > > Some sample data: > > > > | > > {author:"Terry Pratchet",title:"Discworld 1",publisher:"Harper Collins"} > > {author:"Terry Pratchet",title:"Discworld 2",publisher:"Harper Collins"} > > {author:"Terry Pratchet",title:"Discworld 3",publisher:"Harper Collins"} > > {author:"Terry Pratchet",title:"Discworld 4",publisher:"Harper Collins"} > > {author:"Douglas Adams",title:"H2G2",publisher:"Geoffrey Perkins"} > > | > > > > The following request list all publisher, event the deleted ones. > > > > | > > $ curl -XPOST '[http://localhost:9200/books/\_search'-d](http://localhost:9200/books/_search'-d) '{ > > "from":0, > > "size":50, > > "query": { "match\_all":{} }, > > "facets": { > > "publisher": { > > "terms": { > > "field": "publisher", > > "size": 70, > > "all\_terms": true, > > "order": "term" > > } > > } > > }' > > > > | > > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![the\_mouette](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/the_mouette/32/2422_2.png) [@the\_mouette](https://discuss.elastic.co/u/the_mouette)\ **Post date:** [March 22, 2013, 12:15pm UTC](https://discuss.elastic.co/t/facets-with-all-terms-return-deleted-terms/11245/5 "2013-03-22T12:15:22Z")
Hi Clinton, Thanks for your answer, I undersand better how it works internally. I played with optimize, refresh and cache api, and didn't achieve to clean facet queries. curl -XPOST '[http://localhost:9200/\_all/\_optimize](http://localhost:9200/_all/_optimize)' curl -XPOST '[http://localhost:9200/\_all/\_optimize](http://localhost:9200/_all/_optimize)' -d '{max\_num\_segments:1,only\_expunge\_deletes:true,wait\_for\_merge:true}' curl -XPOST '[http://localhost:9200/eliteauto/\_cache/clear](http://localhost:9200/eliteauto/_cache/clear)' curl -XPOST '[http://localhost:9200/\_refresh](http://localhost:9200/_refresh)' all returns expected {"ok":true,"\_shards":{"total":1,"successful":1,"failed":0}} Am I doing something wrong ? Le jeudi 21 mars 2013 17:05:45 UTC+1, Clinton Gormley a écrit : > > I use all\_terms facets and everything went fine until I removed a > > publisher : it is still listed in the facet result. > > The terms list includes terms in documents that have been deleted. > Deleting a document just marks it as deleted and excludes it from future > searches. > > Only once the segment that the document was in is rewritten will the doc > (and the terms) be truly expunged. > > That happens automatically in the background as segments are merged, or > you can force it with the optimize API (which can be quite heavy) > > clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [March 22, 2013, 12:48pm UTC](https://discuss.elastic.co/t/facets-with-all-terms-return-deleted-terms/11245/6 "2013-03-22T12:48:00Z")
Hiya > Thanks for your answer, I undersand better how it works internally. Actually, I appear to have misunderstood how terms facets are calculated. It appears that deleted documents are taken into account. See: [terms.mkd · GitHub](https://gist.github.com/clintongormley/5220985) This at least works on 0.90.0.RC1. I haven't tested it on 0.20\* Which leads me to believe that you still have documents with that publisher. clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![simonw\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simonw_2/32/1130_2.png) [@simonw\_2](https://discuss.elastic.co/u/simonw_2)\ **Post date:** [March 22, 2013, 12:48pm UTC](https://discuss.elastic.co/t/facets-with-all-terms-return-deleted-terms/11245/7 "2013-03-22T12:48:44Z")
looks good to me On Friday, March 22, 2013 1:15:22 PM UTC+1, [the.m...@gmail.com](mailto:the.m...@gmail.com) wrote: > Hi Clinton, > > Thanks for your answer, I undersand better how it works internally. > > I played with optimize, refresh and cache api, and didn't achieve to clean > facet queries. > > curl -XPOST '[http://localhost:9200/\_all/\_optimize](http://localhost:9200/_all/_optimize)' > curl -XPOST '[http://localhost:9200/\_all/\_optimize](http://localhost:9200/_all/_optimize)' -d > '{max\_num\_segments:1,only\_expunge\_deletes:true,wait\_for\_merge:true}' > curl -XPOST '[http://localhost:9200/eliteauto/\_cache/clear](http://localhost:9200/eliteauto/_cache/clear)' > curl -XPOST '[http://localhost:9200/\_refresh](http://localhost:9200/_refresh)' > > all returns expected > > {"ok":true,"\_shards":{"total":1,"successful":1,"failed":0}} > > Am I doing something wrong ? > > Le jeudi 21 mars 2013 17:05:45 UTC+1, Clinton Gormley a écrit : > > > > I use all\_terms facets and everything went fine until I removed a > > > publisher : it is still listed in the facet result. > > > > The terms list includes terms in documents that have been deleted. > > Deleting a document just marks it as deleted and excludes it from future > > searches. > > > > Only once the segment that the document was in is rewritten will the doc > > (and the terms) be truly expunged. > > > > That happens automatically in the background as segments are merged, or > > you can force it with the optimize API (which can be quite heavy) > > > > clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![the\_mouette](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/the_mouette/32/2422_2.png) [@the\_mouette](https://discuss.elastic.co/u/the_mouette)\ **Post date:** [March 22, 2013, 4:16pm UTC](https://discuss.elastic.co/t/facets-with-all-terms-return-deleted-terms/11245/8 "2013-03-22T16:16:36Z")
Indeed ! I thought it might be because of child documents or the custom analyzer, but it isn't either. I've got no clue about what's going on. Just updated to 0.90.0-RC1, and rebuilt the index and it still doesn't work as expected with the real index, whereas it does with the updated gist : [terms.mkd · GitHub](https://gist.github.com/themouette/5221705) The only differences are the _names_, the _number of fields_ (113) and the number of documents (15k for main doc and 1.5M for child) thanks for your answers everybody. Le vendredi 22 mars 2013 13:48:00 UTC+1, Clinton Gormley a écrit : > Hiya > > > Thanks for your answer, I undersand better how it works internally. > > Actually, I appear to have misunderstood how terms facets are > calculated. It appears that deleted documents are taken into account. > > See: [terms.mkd · GitHub](https://gist.github.com/clintongormley/5220985) > > This at least works on 0.90.0.RC1. I haven't tested it on 0.20\* > > Which leads me to believe that you still have documents with that > publisher. > > clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:45am UTC](https://discuss.elastic.co/t/facets-with-all-terms-return-deleted-terms/11245/9 "2017-07-06T02:45:05Z")