# Deleting documents vs deleting the whole index and rebuilding it periodically **URL:** \ **Category:** Elasticsearch\ **Created:** [April 6, 2018, 2:24am UTC](https://discuss.elastic.co/t/deleting-documents-vs-deleting-the-whole-index-and-rebuilding-it-periodically/127018 "2018-04-06T02:24:59Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Hooman\_Bahreini](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hooman_bahreini/32/29500_2.png) [@Hooman\_Bahreini](https://discuss.elastic.co/u/Hooman_Bahreini)\ **Post date:** [April 6, 2018, 2:24am UTC](https://discuss.elastic.co/t/deleting-documents-vs-deleting-the-whole-index-and-rebuilding-it-periodically/127018/1 "2018-04-06T02:24:59Z")
I am creating an online shopping website. Users can create advertisements on the website to sell their products... these advertisements get expired or sold after some time... I am keeping all advertisement in MySQL and copy the advertisement table into ElasticSearch. I want to create a tool as described [here](https://mail.yahoo.com/d/folders/1), to keep ES in sync. Once advertisements expire, I Delete them from ES... so eventually my ES will contain a lot of deleted documents. Is it a good idea to delete the whole Advertisement Index in ES periodically and recreate it with only the active advertisements? Or, does the merging process in ES takes care of deleted documents, and I don't need to worry about deleting and recreating the whole index periodically? Thanks in advance ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [April 6, 2018, 3:51am UTC](https://discuss.elastic.co/t/deleting-documents-vs-deleting-the-whole-index-and-rebuilding-it-periodically/127018/2 "2018-04-06T03:51:28Z")
Elasticsearch takes care of deleted documents and eventually removed them from disk while merging segments. So it's ok in general to remove some documents. But if you are planning to remove a huge number of documents like more than 50-80% of your dataset in one go, then I think it's better to reindex the whole dataset. You can also use time based indices and index documents by their expiration date in dedicated indices like `index-2018-04` and `index-2018-05` for documents expiring in April and May. You can filter out documents you don't want to see anymore with a date range filter. After a month (let's say in May), you can just delete the index `index-2018-04`. That would be efficient IMO. ---
**Author:** ![Hooman\_Bahreini](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hooman_bahreini/32/29500_2.png) [@Hooman\_Bahreini](https://discuss.elastic.co/u/Hooman_Bahreini)\ **Post date:** [April 6, 2018, 5:01am UTC](https://discuss.elastic.co/t/deleting-documents-vs-deleting-the-whole-index-and-rebuilding-it-periodically/127018/3 "2018-04-06T05:01:20Z")
Thanks a lot David. So all advertisement will expire after a month or so... I would probably be deleting less than 1% of the index on each bulk delete request... but after around 1 month, 100% of Index is deleted as all advertisements eventually expire. From your explanation, it sounds like I should be fine with just deleting them as they expire (probably combining a couple of deletes in one bulk delete)? Is my assumption correct that ES merge will happen on a regular basis (say everyday) and therefore this approach could work? I really like your second solution. the problem is I would always have to search the two most recent indices. for example an advertisement created 29th of March, is still active on 6th of April, so I need to search both March and April Indices. Do you think this would work? ---
**Author:** ![nmathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nmathur/32/21451_2.png) [@nmathur](https://discuss.elastic.co/u/nmathur)\ **Post date:** [April 6, 2018, 5:33am UTC](https://discuss.elastic.co/t/deleting-documents-vs-deleting-the-whole-index-and-rebuilding-it-periodically/127018/4 "2018-04-06T05:33:08Z")
We have implemented elasticsearch on a SaaS with millions of users and they are active approx 5-6 hours a day .. I have multiple docs in one index .... we are using delete document procedure .. and we are not facing any problem .. but when ever we stuck to the situation where we need to recreate index .. the process takes much time . because that time we fetch large data to put into elasticsearch. So my suggestion is, please consider delete document. But if you have single doc in single index then may be you can consider deleting the index. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [April 6, 2018, 7:34am UTC](https://discuss.elastic.co/t/deleting-documents-vs-deleting-the-whole-index-and-rebuilding-it-periodically/127018/5 "2018-04-06T07:34:01Z")
> [@Hooman\_Bahreini](#): > > I would probably be deleting less than 1% of the index on each bulk delete request... So that will be fine IMO. > the problem is I would always have to search the two most recent indices. Even though you are probably not going to implement that, let me add some comments. Not really an issue. Just use an alias anytime you create a monthly index. `mydata` -\> `index-2018-04` `mydata` -\> `index-2018-05` Then you just have to search on `mydata` in your application. Elasticsearch will run the search on all indices. Then if you delete the `index-2018-04`, it won't be searched anymore. Anyway, you are not going to use that probably. 🙂 ---
**Author:** ![Hooman\_Bahreini](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hooman_bahreini/32/29500_2.png) [@Hooman\_Bahreini](https://discuss.elastic.co/u/Hooman_Bahreini)\ **Post date:** [April 6, 2018, 7:43am UTC](https://discuss.elastic.co/t/deleting-documents-vs-deleting-the-whole-index-and-rebuilding-it-periodically/127018/6 "2018-04-06T07:43:11Z")
Thanks a lot, David. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 4, 2018, 7:43am UTC](https://discuss.elastic.co/t/deleting-documents-vs-deleting-the-whole-index-and-rebuilding-it-periodically/127018/7 "2018-05-04T07:43:12Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.