# Slowness in Elasticsearch DELETE and UNLOAD operation **URL:** \ **Category:** Elasticsearch\ **Created:** [January 4, 2017, 7:55pm UTC](https://discuss.elastic.co/t/slowness-in-elasticsearch-delete-and-unload-operation/70602 "2017-01-04T19:55:19Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Neuron\_Ring](https://avatars.discourse-cdn.com/v4/letter/n/6a8cbe/32.png) [@Neuron\_Ring](https://discuss.elastic.co/u/Neuron_Ring)\ **Post date:** [January 4, 2017, 7:55pm UTC](https://discuss.elastic.co/t/slowness-in-elasticsearch-delete-and-unload-operation/70602/1 "2017-01-04T19:55:19Z")
Hi,, I have an Elastic cluster with two nodes and about 53 indices. There are two large indices one with 1.5M [size: 93.93MB] documents and the other one has 2.3M [Size 354.61MB] documents in it. Not really huge size.. Not seeing any issues with search or \_bulk PUT operation. **UNLOAD** But when i take a back up of the index using elasticdump it is insanely slow. Takes about 6 hours to complete the unloading of documents using elasticdump. Initially i thought it was because of the write operation. The index receives new documents every 2 mins. So the unload operation never finishes. it keeps unloading every document as the new ones are coming in.. Then, i stopped the scripts which POSTs the new docs. Still it is slow, the Unload takes hours and hours to complete. **DELETE** In addition to unload, i am doing a scheduled maintenance on my indexes everyday. Deleting [using delete\_by\_query API] all the documents older than 60 days. The indexes with smaller size complete in few secs upto to few mins. But the big ones with 1.5M docs in it, is dead slow, took about 4 hours and still running. Am i doing something wrong here? Is it recommended to store millions of documents in one index.. Please advice. It is taking a greater hit on the performance. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 4, 2017, 8:05pm UTC](https://discuss.elastic.co/t/slowness-in-elasticsearch-delete-and-unload-operation/70602/2 "2017-01-04T20:05:41Z")
If you are using time based data, use time based indices, then you just delete any indices older than 60 days. I can't comment on elasticdump though. ---
**Author:** ![Neuron\_Ring](https://avatars.discourse-cdn.com/v4/letter/n/6a8cbe/32.png) [@Neuron\_Ring](https://discuss.elastic.co/u/Neuron_Ring)\ **Post date:** [January 4, 2017, 8:09pm UTC](https://discuss.elastic.co/t/slowness-in-elasticsearch-delete-and-unload-operation/70602/3 "2017-01-04T20:09:32Z")
My index names are not time based. I have one index which has all the data, So i am purging the documents based on "@timestamp" . like this.: range" : { "timestamp" : { "lt" : "now-60d" } ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 4, 2017, 8:10pm UTC](https://discuss.elastic.co/t/slowness-in-elasticsearch-delete-and-unload-operation/70602/4 "2017-01-04T20:10:25Z")
And that's why I am suggesting you change things. ---
**Author:** ![Neuron\_Ring](https://avatars.discourse-cdn.com/v4/letter/n/6a8cbe/32.png) [@Neuron\_Ring](https://discuss.elastic.co/u/Neuron_Ring)\ **Post date:** [January 4, 2017, 8:12pm UTC](https://discuss.elastic.co/t/slowness-in-elasticsearch-delete-and-unload-operation/70602/5 "2017-01-04T20:12:06Z")
So is it an expected behavior of elasticsearch ? Cant it handle millions of documents in one index. ? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 4, 2017, 8:16pm UTC](https://discuss.elastic.co/t/slowness-in-elasticsearch-delete-and-unload-operation/70602/6 "2017-01-04T20:16:49Z")
It can, but DBQ is highly inefficient and there are better ways to logically separate your data to get the same functionality. ---
**Author:** ![Neuron\_Ring](https://avatars.discourse-cdn.com/v4/letter/n/6a8cbe/32.png) [@Neuron\_Ring](https://discuss.elastic.co/u/Neuron_Ring)\ **Post date:** [January 4, 2017, 9:10pm UTC](https://discuss.elastic.co/t/slowness-in-elasticsearch-delete-and-unload-operation/70602/7 "2017-01-04T21:10:54Z")
Okay. Thank you. I like the idea of having the indexes by date thus limiting the data in each index. But the problem is , if i use BULK API to create indexes dynamically, the "string" fields in the mapping gets created as "Analyzed" by default. That will result in creating more storage space allocated for the analysis. So , as initial setup, i create and index and mappings specifying the "Not\_analyzed" and then send the data to it, I dont want any of my "terms" in my mapping to be analyzed. How do i created one index per day having all the fields as "not\_analyzed" ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 5, 2017, 12:13am UTC](https://discuss.elastic.co/t/slowness-in-elasticsearch-delete-and-unload-operation/70602/8 "2017-01-05T00:13:41Z")
> [@Neuron\_Ring](#): > > But the problem is , if i use BULK API to create indexes dynamically, the "string" fields in the mapping gets created as "Analyzed" by default. So use a template that matches your index pattern - [Index Templates | Elasticsearch Guide [5.1] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/5.1/indices-templates.html#indices-templates) 🙂 ---
**Author:** ![Neuron\_Ring](https://avatars.discourse-cdn.com/v4/letter/n/6a8cbe/32.png) [@Neuron\_Ring](https://discuss.elastic.co/u/Neuron_Ring)\ **Post date:** [January 5, 2017, 10:56pm UTC](https://discuss.elastic.co/t/slowness-in-elasticsearch-delete-and-unload-operation/70602/9 "2017-01-05T22:56:31Z")
OK. Thank You ! I would rather like to use the \_rollover api in ES 5.0 ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 5, 2017, 11:07pm UTC](https://discuss.elastic.co/t/slowness-in-elasticsearch-delete-and-unload-operation/70602/10 "2017-01-05T23:07:00Z")
That a great idea! 😃 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 2, 2017, 11:07pm UTC](https://discuss.elastic.co/t/slowness-in-elasticsearch-delete-and-unload-operation/70602/11 "2017-02-02T23:07:16Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.