# Q: Deleting documents by \_id but without index info? **URL:** \ **Category:** Elasticsearch\ **Created:** [October 15, 2014, 1:09pm UTC](https://discuss.elastic.co/t/q-deleting-documents-by--id-but-without-index-info/20265 "2014-10-15T13:09:45Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Bernt\_Rostad](https://avatars.discourse-cdn.com/v4/letter/b/3ab097/32.png) [@Bernt\_Rostad](https://discuss.elastic.co/u/Bernt_Rostad)\ **Post date:** [October 15, 2014, 1:09pm UTC](https://discuss.elastic.co/t/q-deleting-documents-by--id-but-without-index-info/20265/1 "2014-10-15T13:09:45Z")
Hi, This may be a newbie question but I'm struggling to solve a problem for my company in which a MySql database is used to feed and maintain a cluster of Elasticsearch servers used for front-end searches. My problem is: Given a set of document IDs from MySql, I need to remove those documents from all of our Elasticsearch indices. We typically have one index per year and though I know the document ID, which has the same ID in Elasticsearch, I don't know the index it is stored in. I started out trying the Perl module Search::Elasticsearch::Bulk which offered the delete\_ids() and add\_action() methods, which both seemed to allow me to delete a large number of documents, but requiring an index name. I thought I could do the same trick as with the \_search endpoint and use a wildcard to indicate "all indices", e.g. index =\> 'published-\*', but that failed spectacularly: InvalidIndexNameException[[pulse-_] Invalid index name [published-_], must not contain the following characters [, /, \*, ?, ", \<, \>, |, , ,]]; at /usr/local/lib/site\_perl/Cpan/share/perl/5.14.2/Search/Elasticsearch/Role/Bulk.pm line 188. So, I couldn't use a wildcard and I still didn't know the exact index for a given document ID. I was back to the drawing board again. My next attempt was to dynamically decide the index for each document, by doing a search before building up the bulk delete. For this I thought I could use the \_mget endpoint, which seemed similar to \_search and thus would allow me to query all the document IDs to learn their indices. But that didn't work either. Here I've copied the commands I tried running in Sense: GET /published-\*/\_mget { "docs" : [ { "\_id" : "019001201409294aa579ddb20348cbbf402116c91f6d15\_811110" }, { "\_id" : "0190012014092947e9fe351dc78763dabc45231301e9f9\_811110" } ] } GET /\_mget { "docs" : [ { "\_id" : "019001201409294aa579ddb20348cbbf402116c91f6d15\_811110" }, { "\_id" : "0190012014092947e9fe351dc78763dabc45231301e9f9\_811110" } ] } The first only returned errors with each document while the second didn't return anything, just issued an "index is missing" error. However, when calling the \_search endpoint I can either use '/published-\*' or no index info at all and still get a sensible result back, e.g.: GET /\_search { "query": { "filtered": { "query": { "term" : { "\_id" : "0190012014092947e9fe351dc78763dabc45231301e9f9\_811110" } } } } } This has left me perplexed: Why can I query one document ID from the \_search endpoint and get back the index information but not from \_mget? This situation seems to force me to loop over each document ID, of possibly hundreds of thousand per night, calling the \_search endpoint for each ID to get the index information and then build up the bulk delete. Can life really be this difficult? Are there other mechanisms I can look at that will allow me, for a given list of document IDs, to delete the associated documents from unspecified Elasticsearch indices? I'm sorry if this was a trivial question but I've spent several days pouring over the Search::Elasticsearch documentation and googling Elasticsearch examples without finding other ways to get the job done. Best wishes, Bernt Rostad Retriever Norge -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/719fd493-9eaa-4f22-8552-3f1c91245786%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/719fd493-9eaa-4f22-8552-3f1c91245786%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Bernt\_Rostad](https://avatars.discourse-cdn.com/v4/letter/b/3ab097/32.png) [@Bernt\_Rostad](https://discuss.elastic.co/u/Bernt_Rostad)\ **Post date:** [October 16, 2014, 10:17am UTC](https://discuss.elastic.co/t/q-deleting-documents-by--id-but-without-index-info/20265/2 "2014-10-16T10:17:30Z")
Found the answer after another day of trial & errors. For those interested, the clue is the "ids" parameter in a regular \_search query: GET /published-\*/\_search { "query" : { "ids" : { "values" : ["019001201409294aa579ddb20348cbbf402116c91f6d15\_811110", "0190012014092947e9fe351dc78763dabc45231301e9f9\_811110"] } } } This allows me to fetch index data for for each of the document IDs 🙂 bernt On Wednesday, October 15, 2014 3:09:45 PM UTC+2, Bernt Rostad wrote: > Hi, > > This may be a newbie question but I'm struggling to solve a problem for my > company in which a MySql database is used to feed and maintain a cluster of > Elasticsearch servers used for front-end searches. > > My problem is: Given a set of document IDs from MySql, I need to remove > those documents from all of our Elasticsearch indices. We typically have > one index per year and though I know the document ID, which has the same ID > in Elasticsearch, I don't know the index it is stored in. > > I started out trying the Perl module Search::Elasticsearch::Bulk which > offered the delete\_ids() and add\_action() methods, which both seemed to > allow me to delete a large number of documents, but requiring an index > name. I thought I could do the same trick as with the \_search endpoint and > use a wildcard to indicate "all indices", e.g. index =\> 'published-\*', but > that failed spectacularly: > > InvalidIndexNameException[[pulse-_] Invalid index name [published-_], must > not contain the following characters [, /, \*, ?, ", \<, \>, |, , ,]]; at > /usr/local/lib/site\_perl/Cpan/share/perl/5.14.2/Search/Elasticsearch/Role/Bulk.pm > line 188. > > So, I couldn't use a wildcard and I still didn't know the exact index for > a given document ID. I was back to the drawing board again. > > My next attempt was to dynamically decide the index for each document, by > doing a search before building up the bulk delete. For this I thought I > could use the \_mget endpoint, which seemed similar to \_search and thus > would allow me to query all the document IDs to learn their indices. But > that didn't work either. Here I've copied the commands I tried running in > Sense: > > GET /published-\*/\_mget > { > "docs" : [ > { "\_id" : "019001201409294aa579ddb20348cbbf402116c91f6d15\_811110" }, > { "\_id" : "0190012014092947e9fe351dc78763dabc45231301e9f9\_811110" } > ] > } > > GET /\_mget > { > "docs" : [ > { "\_id" : "019001201409294aa579ddb20348cbbf402116c91f6d15\_811110" }, > { "\_id" : "0190012014092947e9fe351dc78763dabc45231301e9f9\_811110" } > ] > } > > The first only returned errors with each document while the second didn't > return anything, just issued an "index is missing" error. > > However, when calling the \_search endpoint I can either use '/published-\*' > or no index info at all and still get a sensible result back, e.g.: > > GET /\_search > { > "query": { > "filtered": { > "query": { > "term" : { "\_id" : > "0190012014092947e9fe351dc78763dabc45231301e9f9\_811110" } > } > } > } > } > > This has left me perplexed: Why can I query one document ID from the > \_search endpoint and get back the index information but not from \_mget? > > This situation seems to force me to loop over each document ID, of > possibly hundreds of thousand per night, calling the \_search endpoint for > each ID to get the index information and then build up the bulk delete. > > Can life really be this difficult? > > Are there other mechanisms I can look at that will allow me, for a given > list of document IDs, to delete the associated documents from unspecified > Elasticsearch indices? > > I'm sorry if this was a trivial question but I've spent several days > pouring over the Search::Elasticsearch documentation and googling > Elasticsearch examples without finding other ways to get the job done. > > Best wishes, > Bernt Rostad > Retriever Norge -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/bce09ee4-9742-46ac-9c98-7ce274b95818%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/bce09ee4-9742-46ac-9c98-7ce274b95818%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:55am UTC](https://discuss.elastic.co/t/q-deleting-documents-by--id-but-without-index-info/20265/3 "2017-07-06T00:55:48Z")