# Elastic hard delete
**URL:** https://discuss.elastic.co/t/elastic-hard-delete/156432
**Category:** Elasticsearch
**Created:** [November 13, 2018, 9:57am UTC](https://discuss.elastic.co/t/elastic-hard-delete/156432 "2018-11-13T09:57:21Z")
**Posts on this page:** 14
**Page:** 1
### Author:  [@nileshsoor](https://discuss.elastic.co/u/nileshsoor)
#### Post date: [November 13, 2018, 9:57am UTC](https://discuss.elastic.co/t/elastic-hard-delete/156432/1 "2018-11-13T09:57:21Z")
Hello,
I have created an Index of 200M docs and will be updated frequently. 50M docs will be updated in a month. In short, the Index is read/write heavy in nature.
As Elastic/Lucene says it will not do an actual update but it will delete/add, that means the deleted docs will resides in Index but will not be searchable.
Lucene occasionally _merges_ segments according to merge policy, which is costlier.
So my question is
1. How would be my Index read/write performant in such scenario?
2. Is there any alternative to deal with such scenario?
3. Can we disable soft delete in Elastic/Lucene and allow only hard delete?
Thanks,
Nilesh
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [November 13, 2018, 10:17am UTC](https://discuss.elastic.co/t/elastic-hard-delete/156432/2 "2018-11-13T10:17:52Z")
> [@nileshsoor](#):
>
> I have created an Index of 200M docs and will be updated frequently.
How often is frequently? Which version of Elasticsearch are you using?
> [@nileshsoor](#):
>
> As Elastic/Lucene says it will not do an actual update but it will delete/add, that means the deleted docs will resides in Index but will not be searchable.
Correct. They will only be physically deleted from disk during a merge.
> [@nileshsoor](#):
>
> How would be my Index read/write performant in such scenario?
It depends on the use case and the questions I asked earlier.
> [@nileshsoor](#):
>
> Is there any alternative to deal with such scenario?
I am not sure I understand your question.
> [@nileshsoor](#):
>
> Can we disable soft delete in Elastic/Lucene and allow only hard delete?
No, that is not possible.
---
### Author:  [@nileshsoor](https://discuss.elastic.co/u/nileshsoor)
#### Post date: [November 13, 2018, 10:28am UTC](https://discuss.elastic.co/t/elastic-hard-delete/156432/3 "2018-11-13T10:28:51Z")
> [@Christian\_Dahlqvist](#):
>
> How often is frequently? Which version of Elasticsearch are you using?
Version : 5.4
In a month,
1. for 7 days bulk update 30-35M (from data source 1)
2. daily 1M bulk update. (from data source 2..n)
3. +Heavy search
---
### Author:  [@nik9000](https://discuss.elastic.co/u/nik9000)
#### Post date: [November 13, 2018, 10:30am UTC](https://discuss.elastic.co/t/elastic-hard-delete/156432/4 "2018-11-13T10:30:12Z")
The only way to not use the delete-is-mark-then-merge-away behavior is to create an entirely new index and then drop the old one. Some people do this, but mostly because they don't have triggers to sync changes so they must so it periodically.
I ran a system that had many updates. Performance was fine. The standard advice of setting the refresh time to 30 seconds if you can tolerate it is good here. As is watching updates and dedupicating them.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [November 13, 2018, 10:32am UTC](https://discuss.elastic.co/t/elastic-hard-delete/156432/5 "2018-11-13T10:32:01Z")
What is the problem you are trying to solve?
---
### Author:  [@nileshsoor](https://discuss.elastic.co/u/nileshsoor)
#### Post date: [November 13, 2018, 10:38am UTC](https://discuss.elastic.co/t/elastic-hard-delete/156432/6 "2018-11-13T10:38:13Z")
> [@Christian\_Dahlqvist](#):
>
> What is the problem you are trying to solve?
We are building a system that takes(update) data from multiple sources and update the master data. At the same time, consumer of the master should be able to get the updated data from master real time.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [November 13, 2018, 10:38am UTC](https://discuss.elastic.co/t/elastic-hard-delete/156432/7 "2018-11-13T10:38:55Z")
Are you seeing any performance problems you are trying to address?
---
### Author:  [@nileshsoor](https://discuss.elastic.co/u/nileshsoor)
#### Post date: [November 13, 2018, 10:49am UTC](https://discuss.elastic.co/t/elastic-hard-delete/156432/8 "2018-11-13T10:49:59Z")
> [@Christian\_Dahlqvist](#):
>
> Are you seeing any performance problems you are trying to address?
Did POC using following configuration
3 node cluster
RAM : 60gb
HD : 1.5 TB SSD
CPU : 8 core
POC :
Indexing and searching(fuzzy query) simultaneously in multi-threaded environment.
What I observed is, Indexing slows down drastically. If I do only indexing then the performance is very good.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [November 13, 2018, 10:55am UTC](https://discuss.elastic.co/t/elastic-hard-delete/156432/9 "2018-11-13T10:55:36Z")
What was limiting performance when you indexed and searched at the same time? CPU? Disk I/O? Did you see any reports in the logs about long or slow GC?
---
### Author:  [@nileshsoor](https://discuss.elastic.co/u/nileshsoor)
#### Post date: [November 13, 2018, 11:05am UTC](https://discuss.elastic.co/t/elastic-hard-delete/156432/10 "2018-11-13T11:05:48Z")
> [@nik9000](#):
>
> The only way to not use the delete-is-mark-then-merge-away behavior is to create an entirely new index and then drop the old one. Some people do this, but mostly because they don't have triggers to sync changes so they must so it periodically.
>
> I ran a system that had many updates. Performance was fine. The standard advice of setting the refresh time to 30 seconds if you can tolerate it is good here. As is watching updates and dedupicating them.
How you handled deleted docs?
Are you relying on Elastic/Lucene to purge?
What settings you used?
---
### Author:  [@nileshsoor](https://discuss.elastic.co/u/nileshsoor)
#### Post date: [November 13, 2018, 11:09am UTC](https://discuss.elastic.co/t/elastic-hard-delete/156432/11 "2018-11-13T11:09:30Z")
> [@Christian\_Dahlqvist](#):
>
> What was limiting performance when you indexed and searched at the same time? CPU? Disk I/O? Did you see any reports in the logs about long or slow GC?
Using htop, got to know CPU & RAM.
Even if I increased the infrastructure, purging will not affect performance?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [November 13, 2018, 11:14am UTC](https://discuss.elastic.co/t/elastic-hard-delete/156432/12 "2018-11-13T11:14:28Z")
Indexing, merging and querying use the same system resources, so off course they can affect performance and each other. You need to test to see that you have enough system resources available for the combined load and that you at that point are able to index and query with acceptable performance.
---
### Author:  [@nileshsoor](https://discuss.elastic.co/u/nileshsoor)
#### Post date: [November 13, 2018, 11:23am UTC](https://discuss.elastic.co/t/elastic-hard-delete/156432/13 "2018-11-13T11:23:17Z")
Thanks Christian\_Dahlqvist & nik9000!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 11, 2018, 11:36am UTC](https://discuss.elastic.co/t/elastic-hard-delete/156432/14 "2018-12-11T11:36:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.