# Can't get Documents Deleted below 40% - performance issues - help needed **URL:** \ **Category:** Elasticsearch\ **Created:** [February 1, 2015, 7:31pm UTC](https://discuss.elastic.co/t/cant-get-documents-deleted-below-40-performance-issues-help-needed/21942 "2015-02-01T19:31:16Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![theplastictoy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theplastictoy/32/44915_2.png) [@theplastictoy](https://discuss.elastic.co/u/theplastictoy)\ **Post date:** [February 1, 2015, 7:31pm UTC](https://discuss.elastic.co/t/cant-get-documents-deleted-below-40-performance-issues-help-needed/21942/1 "2015-02-01T19:31:16Z")
Hi, We've been using ElasticSearch (through qBox) for almost 3 months now, but something's very wrong and we need help on this one. I'll try to give as much context as possible. Last month we had a major downtime (almost 24hr). We were using a single node with 2GB and after this downtime we upgraded to a 3 node 4GB setup. We are online again, but our Documents Deleted values are still way too high (40-50% and more) and they never go down, which leaves us uncomfortable. We've contacted qBox, which are very heplful, but they always say we need a bigger node because of that. It's really not an option, as hosting costs of our product skyrocketed with this. Our product is an invoicing platform. Our text search, filters and all invoice listings now get all their data from elastic instead of MySQL, so we can't rely (we think) on bulk indexing, as the invoices, items, clients, etc.. need to be indexed asap to be shown in listings. Maybe we have a design issue and shouldn't be using elastic for so many things, it's something we need to understand better. At the moment, our main concern in making sure we're ok in the long run and avoid another huge downtime at all costs. Can someone help in some way? We'd be very grateful. Some more info: - We're still not using round robbin. Could this help? - Version 1.2.4. It's on our roadmap to upgrade but we've heard stories of worst performance in some cases (groovy-related). - I've added some screens with node info from qbox. Thanks, Ricardo -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/eca7aa05-fda6-4c21-9fff-dd418d6ae2cb%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/eca7aa05-fda6-4c21-9fff-dd418d6ae2cb%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 1, 2015, 8:50pm UTC](https://discuss.elastic.co/t/cant-get-documents-deleted-below-40-performance-issues-help-needed/21942/2 "2015-02-01T20:50:51Z")
If you're updating single documents often then you should expect high delete rates, your heap and CPU use seems to be ok though so it's not anything to be super concerned with (for now). You do have the option of forcing an optimise (which does a merge and removes deleted docs), but this is resource intensive so be careful of when you run it, ie it'd be best to schedule it to run when your users are asleep. Your main indices seem to only have 2 shards, with no replicas. You should really have one shard per node, with one replica to give you some redundancy for a total of 6 shards, 3 primary, 3 replica. Currently you will have an imbalance of data, which means some nodes are more overloaded than others, plus without replicas you cannot survive the loss of a node. Without knowing your use, it also looks like something is creating indices that shouldn't be there - eg phppath, cgi-bin, cfg, phpmyadmin - which doesn't look like a problem at the moment, but is something you should check, out if they shouldn't be there that is). Finally, if you want better performance, move to Oracle JDK and then upgrade to the latest release (1.4.3). There is a lot of improvements in the latest versions, though I cannot comment on the Groovy side of your question. On 2 February 2015 at 06:31, Ricardo Fiel [ricardo.fiel@rupeal.com](mailto:ricardo.fiel@rupeal.com) wrote: > Hi, > > We've been using Elasticsearch (through qBox) for almost 3 months now, but > something's very wrong and we need help on this one. I'll try to give as > much context as possible. > > Last month we had a major downtime (almost 24hr). We were using a single > node with 2GB and after this downtime we upgraded to a 3 node 4GB setup. We > are online again, but our Documents Deleted values are still way too high > (40-50% and more) and they never go down, which leaves us uncomfortable. > We've contacted qBox, which are very heplful, but they always say we need a > bigger node because of that. It's really not an option, as hosting costs of > our product skyrocketed with this. > > Our product is an invoicing platform. Our text search, filters and all > invoice listings now get all their data from elastic instead of MySQL, so > we can't rely (we think) on bulk indexing, as the invoices, items, clients, > etc.. need to be indexed asap to be shown in listings. Maybe we have a > design issue and shouldn't be using elastic for so many things, it's > something we need to understand better. > > At the moment, our main concern in making sure we're ok in the long run > and avoid another huge downtime at all costs. > > Can someone help in some way? We'd be very grateful. > > Some more info: > > - We're still not using round robbin. Could this help? > - Version 1.2.4. It's on our roadmap to upgrade but we've heard > stories of worst performance in some cases (groovy-related). > - I've added some screens with node info from qbox. > > Thanks, > Ricardo > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/eca7aa05-fda6-4c21-9fff-dd418d6ae2cb%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/eca7aa05-fda6-4c21-9fff-dd418d6ae2cb%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/eca7aa05-fda6-4c21-9fff-dd418d6ae2cb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/eca7aa05-fda6-4c21-9fff-dd418d6ae2cb%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-L-C%3DSfNjRN6559%3DGsQ90qY\_HsugOmV-i2Di3vOf8r1w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-L-C%3DSfNjRN6559%3DGsQ90qY_HsugOmV-i2Di3vOf8r1w%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![mikemccand](https://avatars.discourse-cdn.com/v4/letter/m/f04885/32.png) [@mikemccand](https://discuss.elastic.co/u/mikemccand)\ **Post date:** [February 1, 2015, 9:05pm UTC](https://discuss.elastic.co/t/cant-get-documents-deleted-below-40-performance-issues-help-needed/21942/3 "2015-02-01T21:05:28Z")
It's normal to see 40-60% deleted docs if you frequently update existing documents. See this recent blog post I wrote for some details: > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. Mike McCandless [http://blog.mikemccandless.com](http://blog.mikemccandless.com) On Sun, Feb 1, 2015 at 3:50 PM, Mark Walkom [markwalkom@gmail.com](mailto:markwalkom@gmail.com) wrote: > If you're updating single documents often then you should expect high > delete rates, your heap and CPU use seems to be ok though so it's not > anything to be super concerned with (for now). You do have the option of > forcing an optimise (which does a merge and removes deleted docs), but this > is resource intensive so be careful of when you run it, ie it'd be best to > schedule it to run when your users are asleep. > > Your main indices seem to only have 2 shards, with no replicas. You should > really have one shard per node, with one replica to give you some > redundancy for a total of 6 shards, 3 primary, 3 replica. Currently you > will have an imbalance of data, which means some nodes are more overloaded > than others, plus without replicas you cannot survive the loss of a node. > > Without knowing your use, it also looks like something is creating indices > that shouldn't be there - eg phppath, cgi-bin, cfg, phpmyadmin - which > doesn't look like a problem at the moment, but is something you should > check, out if they shouldn't be there that is). > > Finally, if you want better performance, move to Oracle JDK and then > upgrade to the latest release (1.4.3). There is a lot of improvements in > the latest versions, though I cannot comment on the Groovy side of your > question. > > On 2 February 2015 at 06:31, Ricardo Fiel [ricardo.fiel@rupeal.com](mailto:ricardo.fiel@rupeal.com) wrote: > > > Hi, > > > > We've been using Elasticsearch (through qBox) for almost 3 months now, > > but something's very wrong and we need help on this one. I'll try to give > > as much context as possible. > > > > Last month we had a major downtime (almost 24hr). We were using a single > > node with 2GB and after this downtime we upgraded to a 3 node 4GB setup. We > > are online again, but our Documents Deleted values are still way too high > > (40-50% and more) and they never go down, which leaves us uncomfortable. > > We've contacted qBox, which are very heplful, but they always say we need a > > bigger node because of that. It's really not an option, as hosting costs of > > our product skyrocketed with this. > > > > Our product is an invoicing platform. Our text search, filters and all > > invoice listings now get all their data from elastic instead of MySQL, so > > we can't rely (we think) on bulk indexing, as the invoices, items, clients, > > etc.. need to be indexed asap to be shown in listings. Maybe we have a > > design issue and shouldn't be using elastic for so many things, it's > > something we need to understand better. > > > > At the moment, our main concern in making sure we're ok in the long run > > and avoid another huge downtime at all costs. > > > > Can someone help in some way? We'd be very grateful. > > > > Some more info: > > > > - We're still not using round robbin. Could this help? > > - Version 1.2.4. It's on our roadmap to upgrade but we've heard > > stories of worst performance in some cases (groovy-related). > > - I've added some screens with node info from qbox. > > > > Thanks, > > Ricardo > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/eca7aa05-fda6-4c21-9fff-dd418d6ae2cb%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/eca7aa05-fda6-4c21-9fff-dd418d6ae2cb%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/eca7aa05-fda6-4c21-9fff-dd418d6ae2cb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/eca7aa05-fda6-4c21-9fff-dd418d6ae2cb%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-L-C%3DSfNjRN6559%3DGsQ90qY\_HsugOmV-i2Di3vOf8r1w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-L-C%3DSfNjRN6559%3DGsQ90qY_HsugOmV-i2Di3vOf8r1w%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-L-C%3DSfNjRN6559%3DGsQ90qY\_HsugOmV-i2Di3vOf8r1w%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-L-C%3DSfNjRN6559%3DGsQ90qY_HsugOmV-i2Di3vOf8r1w%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAD7smRcJKMYKg-OX2CkpGezF9VDNYpXOE0mvewFU112j1yXSNg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAD7smRcJKMYKg-OX2CkpGezF9VDNYpXOE0mvewFU112j1yXSNg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![theplastictoy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theplastictoy/32/44915_2.png) [@theplastictoy](https://discuss.elastic.co/u/theplastictoy)\ **Post date:** [February 1, 2015, 11:18pm UTC](https://discuss.elastic.co/t/cant-get-documents-deleted-below-40-performance-issues-help-needed/21942/4 "2015-02-01T23:18:04Z")
Hi Mark, hi Mike, Thanks both for helping on this. The article's great, I didn't know about the 40-60 sawtooth pattern. So, I guess this means we should be ok for now as long as we monitor CPU usage, correct? Don't have access to the server OS (that's on qBox side), so moving to Oracle JDK is not an option. I thought the 3 node set-up is the recommended, so when a node goes down, the cluster could recover automatically, but I was scratching my head on why there were no replicas. I guess your answer made it clear. I need to ask qBox this. Next step is changing the app to use round robbin, that may distribute the load a bit better. Thanks, Ricardo On Sunday, February 1, 2015 at 9:06:04 PM UTC, Michael McCandless wrote: > It's normal to see 40-60% deleted docs if you frequently update existing > documents. See this recent blog post I wrote for some details: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/blog/lucenes-handling-of-deleted-documents/) > > Mike McCandless > > [http://blog.mikemccandless.com](http://blog.mikemccandless.com) > > On Sun, Feb 1, 2015 at 3:50 PM, Mark Walkom \<[markw...@gmail.com](mailto:markw...@gmail.com) > \\> wrote: > > > If you're updating single documents often then you should expect high > > delete rates, your heap and CPU use seems to be ok though so it's not > > anything to be super concerned with (for now). You do have the option of > > forcing an optimise (which does a merge and removes deleted docs), but this > > is resource intensive so be careful of when you run it, ie it'd be best to > > schedule it to run when your users are asleep. > > > > Your main indices seem to only have 2 shards, with no replicas. You > > should really have one shard per node, with one replica to give you some > > redundancy for a total of 6 shards, 3 primary, 3 replica. Currently you > > will have an imbalance of data, which means some nodes are more overloaded > > than others, plus without replicas you cannot survive the loss of a node. > > > > Without knowing your use, it also looks like something is creating > > indices that shouldn't be there - eg phppath, cgi-bin, cfg, phpmyadmin - > > which doesn't look like a problem at the moment, but is something you > > should check, out if they shouldn't be there that is). > > > > Finally, if you want better performance, move to Oracle JDK and then > > upgrade to the latest release (1.4.3). There is a lot of improvements in > > the latest versions, though I cannot comment on the Groovy side of your > > question. > > > > On 2 February 2015 at 06:31, Ricardo Fiel \<[ricard...@rupeal.com](mailto:ricard...@rupeal.com) > > \\> wrote: > > > > > Hi, > > > > > > We've been using Elasticsearch (through qBox) for almost 3 months now, > > > but something's very wrong and we need help on this one. I'll try to give > > > as much context as possible. > > > > > > Last month we had a major downtime (almost 24hr). We were using a single > > > node with 2GB and after this downtime we upgraded to a 3 node 4GB setup. We > > > are online again, but our Documents Deleted values are still way too high > > > (40-50% and more) and they never go down, which leaves us uncomfortable. > > > We've contacted qBox, which are very heplful, but they always say we need a > > > bigger node because of that. It's really not an option, as hosting costs of > > > our product skyrocketed with this. > > > > > > Our product is an invoicing platform. Our text search, filters and all > > > invoice listings now get all their data from elastic instead of MySQL, so > > > we can't rely (we think) on bulk indexing, as the invoices, items, clients, > > > etc.. need to be indexed asap to be shown in listings. Maybe we have a > > > design issue and shouldn't be using elastic for so many things, it's > > > something we need to understand better. > > > > > > At the moment, our main concern in making sure we're ok in the long run > > > and avoid another huge downtime at all costs. > > > > > > Can someone help in some way? We'd be very grateful. > > > > > > Some more info: > > > > > > - We're still not using round robbin. Could this help? > > > - Version 1.2.4. It's on our roadmap to upgrade but we've heard > > > stories of worst performance in some cases (groovy-related). > > > - I've added some screens with node info from qbox. > > > > > > Thanks, > > > Ricardo > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/eca7aa05-fda6-4c21-9fff-dd418d6ae2cb%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/eca7aa05-fda6-4c21-9fff-dd418d6ae2cb%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/eca7aa05-fda6-4c21-9fff-dd418d6ae2cb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/eca7aa05-fda6-4c21-9fff-dd418d6ae2cb%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-L-C%3DSfNjRN6559%3DGsQ90qY\_HsugOmV-i2Di3vOf8r1w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-L-C%3DSfNjRN6559%3DGsQ90qY_HsugOmV-i2Di3vOf8r1w%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-L-C%3DSfNjRN6559%3DGsQ90qY\_HsugOmV-i2Di3vOf8r1w%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-L-C%3DSfNjRN6559%3DGsQ90qY_HsugOmV-i2Di3vOf8r1w%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c524e7fc-fea4-46cb-bf8c-ab1bfe2f11c3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c524e7fc-fea4-46cb-bf8c-ab1bfe2f11c3%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:35am UTC](https://discuss.elastic.co/t/cant-get-documents-deleted-below-40-performance-issues-help-needed/21942/5 "2017-07-06T00:35:26Z")