# ElasticSearch deletes documents in an index automatically **URL:** \ **Category:** Elasticsearch\ **Created:** [November 7, 2022, 10:51am UTC](https://discuss.elastic.co/t/elasticsearch-deletes-documents-in-an-index-automatically/318316 "2022-11-07T10:51:16Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![joanjanku2000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joanjanku2000/32/104542_2.png) [@joanjanku2000](https://discuss.elastic.co/u/joanjanku2000)\ **Post date:** [November 7, 2022, 10:51am UTC](https://discuss.elastic.co/t/elasticsearch-deletes-documents-in-an-index-automatically/318316/1 "2022-11-07T10:51:16Z")
I have configured an **ELK Cluster** with 5 nodes, one being master and the other slaves. I index logs in the cluster once a day using logstash. I use a **CronJOB** (script) to copy the log files to the **configured logstash directory**. I have also **manually** set a **.sincedb path for logstash**. However, a tricky thing happens. Almost every 3 days, **index seems to be loosing documents and deleting everything prior to certain dates**. **I haven't configured any ILM policy** , nor there is any script performing delete by query or delete full index. Even when calling \_cat/indices formatted to show the creation date of te index, I see that it has been created almost 2 weeks ago. However, the documents that should've been for 2 weeks aren't there anymore, and even today it only had documents from 3 days ago. Does anyone know why could this behaviour be happening or what can trigger it ? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 7, 2022, 11:25am UTC](https://discuss.elastic.co/t/elasticsearch-deletes-documents-in-an-index-automatically/318316/2 "2022-11-07T11:25:10Z")
> [@joanjanku2000](#): > > I have configured an **ELK Cluster** with 5 nodes, one being master and the other slaves. Elasticsearch does not have master and slaves - the nodes form a cluster. Because of this you should always look to have 3 master eligible nodes in the cluster as the single master eligible node otherwise is a single point of failure. There is nothing in Elasticsearch that delete data in indices, and ILM deletes complete indices. It would help if you shared your Logstash pipeline as it could be an issue with this that causes this behaviour, e.g. if you are incorrectly etting document IDs in your pipeline. ---
**Author:** ![joanjanku2000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joanjanku2000/32/104542_2.png) [@joanjanku2000](https://discuss.elastic.co/u/joanjanku2000)\ **Post date:** [November 7, 2022, 11:29am UTC](https://discuss.elastic.co/t/elasticsearch-deletes-documents-in-an-index-automatically/318316/3 "2022-11-07T11:29:03Z")
> input { > file { > path =\> "/home/somedir/haproxy/_.log" > type =\> "haproxy" > } > file { > path =\> "/home/somedir/_.log" > type =\> "webmap" > } > } > > filter { > if [type] == "haproxy" { > grok { > match =\> ["message", "%{NOTSPACE:month}%{SPACE:sp}%{NOTSPACE:day} %{NOTSPACE:time} %{NOTSPACE:smth} %{NOTSPACE:process\_and\_id} %{IP:client\_ip}:%{NUMBER:client\_port:int} [%{NOTSPACE:haproxy\_times > tamp}] %{NOTSPACE:frontend\_name} %{NOTSPACE:backend\_name}/%{NOTSPACE:server\_name} %{NUMBER:TRequest:int}/%{NUMBER:TQueues:int}/%{NUMBER:TConnectionInServer:int}/%{NUMBER:TResponseFromServer:int}/%{NUMBER:TR > equestActiveInHaproxy:int} %{NUMBER:status\_code} %{NUMBER:bytes\_read:int} %{NOTSPACE:captured\_request\_cookie} %{NOTSPACE:captured\_response\_cookie} %{NOTSPACE:termination\_state} %{NOTSPACE:some\_flags} %{NOTS > PACE:srv\_queue}/%{NOTSPACE:backend\_queue} {%{HAPROXYCAPTUREDREQUESTHEADERS}} "%{NOTSPACE:http\_verb} %{NOTSPACE:http\_host}?%{NOTSPACE:http\_params} %{NOTSPACE:http\_protocol}"" > , > "message", "%{NOTSPACE:month}%{SPACE:sp}%{NOTSPACE:day} %{NOTSPACE:time} %{NOTSPACE:smth} %{NOTSPACE:process\_and\_id} %{IP:client\_ip}:%{NUMBER:client\_port:int} [%{NOTSPACE:haproxy\_timest > amp}] %{NOTSPACE:frontend\_name} %{NOTSPACE:backend\_name}/%{NOTSPACE:server\_name} %{NUMBER:TRequest:int}/%{NUMBER:TQueues:int}/%{NUMBER:TConnectionInServer:int}/%{NUMBER:TResponseFromServer:int}/%{NUMBER:TRe > questActiveInHaproxy:int} %{NUMBER:status\_code} %{NUMBER:bytes\_read:int} %{NOTSPACE:captured\_request\_cookie} %{NOTSPACE:captured\_response\_cookie} %{NOTSPACE:termination\_state} %{NOTSPACE:some\_flags} %{NOTSP > ACE:srv\_queue}/%{NOTSPACE:backend\_queue} {%{HAPROXYCAPTUREDREQUESTHEADERS}} "%{NOTSPACE:http\_verb} %{NOTSPACE:http\_host} %{NOTSPACE:http\_protocol}"" > ] > } > date { > match =\> ["haproxy\_timestamp","dd/MMM/yyyy:HH:mm:ss.SSS"] > target =\> "@timestamp" > } > } else if [type] == "webmap" { > grok { > match =\> {"message" =\> "(?%{DATE\_EU} %{TIME}) %{WORD:timezoneContinent}/%{WORD:timezoneState} %{LOGLEVEL:loglevel} %{WORD:logername} - %{WORD:user}--%{DATA:action}--%{WORD:entit > y\_name}--%{WORD:id}--%{WORD:methodName}" } > > ``` > } > date { > match => ["timestamp","dd.MM.yyyy HH:mm:ss"] > target => "@timestamp" > } > } > > ``` > > } > > output { > if [type] == "haproxy" { > elasticsearch { > hosts =\> ["192.168.X.XXX:9200"] > index =\> "logs\_haproxy" > } > } > else if [type] == "webmap" { > elasticsearch { > hosts =\> ["192.168.X.XXX:9200"] > index =\> "digital\_footprint\_logs" > } > } > } Hi , yes I am attaching the logstash pipeline. Could file deletion from the input directory, cause docs to be deleted ? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 7, 2022, 11:29am UTC](https://discuss.elastic.co/t/elasticsearch-deletes-documents-in-an-index-automatically/318316/4 "2022-11-07T11:29:58Z")
Please do not post screenshots of text - it can be very hard to read. Instead copy and paste and format using the tools available. It seems like you are not setting any ID so that means the Logstash pipeline is likely not the problem. I can however see that you have not secured your cluster. That means someone could access and delete data without your knowledge. I would recommend securing it. ---
**Author:** ![joanjanku2000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joanjanku2000/32/104542_2.png) [@joanjanku2000](https://discuss.elastic.co/u/joanjanku2000)\ **Post date:** [November 7, 2022, 11:32am UTC](https://discuss.elastic.co/t/elasticsearch-deletes-documents-in-an-index-automatically/318316/5 "2022-11-07T11:32:31Z")
Yes, just edited the response above ---
**Author:** ![joanjanku2000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joanjanku2000/32/104542_2.png) [@joanjanku2000](https://discuss.elastic.co/u/joanjanku2000)\ **Post date:** [November 7, 2022, 11:33am UTC](https://discuss.elastic.co/t/elasticsearch-deletes-documents-in-an-index-automatically/318316/6 "2022-11-07T11:33:40Z")
Yes, I am in the process of working with security in the test environment. Thanks for the sugestion. ---
**Author:** ![joanjanku2000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joanjanku2000/32/104542_2.png) [@joanjanku2000](https://discuss.elastic.co/u/joanjanku2000)\ **Post date:** [November 7, 2022, 11:34am UTC](https://discuss.elastic.co/t/elasticsearch-deletes-documents-in-an-index-automatically/318316/7 "2022-11-07T11:34:23Z")
It's strange cause it works perfectly fine in test env. This problem only occurs in prod ---
**Author:** ![joanjanku2000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joanjanku2000/32/104542_2.png) [@joanjanku2000](https://discuss.elastic.co/u/joanjanku2000)\ **Post date:** [November 7, 2022, 11:35am UTC](https://discuss.elastic.co/t/elasticsearch-deletes-documents-in-an-index-automatically/318316/8 "2022-11-07T11:35:31Z")
And I also use runtime mapped fields. So the index being deleted and re-created would also need runtime mapped fields to be configured through painless script which can only be done by some bash scripts I have, and are not in any crontabs. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 7, 2022, 11:35am UTC](https://discuss.elastic.co/t/elasticsearch-deletes-documents-in-an-index-automatically/318316/9 "2022-11-07T11:35:40Z")
Which version of Elasticsearch are you using? ---
**Author:** ![joanjanku2000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joanjanku2000/32/104542_2.png) [@joanjanku2000](https://discuss.elastic.co/u/joanjanku2000)\ **Post date:** [November 7, 2022, 11:36am UTC](https://discuss.elastic.co/t/elasticsearch-deletes-documents-in-an-index-automatically/318316/10 "2022-11-07T11:36:10Z")
Meaning that the index is not likely deleted but rather the specific documents are being deleted ---
**Author:** ![joanjanku2000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joanjanku2000/32/104542_2.png) [@joanjanku2000](https://discuss.elastic.co/u/joanjanku2000)\ **Post date:** [November 7, 2022, 11:36am UTC](https://discuss.elastic.co/t/elasticsearch-deletes-documents-in-an-index-automatically/318316/11 "2022-11-07T11:36:25Z")
It's 7.17 ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 7, 2022, 11:36am UTC](https://discuss.elastic.co/t/elasticsearch-deletes-documents-in-an-index-automatically/318316/12 "2022-11-07T11:36:48Z")
That means that some external process is likely deleting them. ---
**Author:** ![joanjanku2000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joanjanku2000/32/104542_2.png) [@joanjanku2000](https://discuss.elastic.co/u/joanjanku2000)\ **Post date:** [November 7, 2022, 11:37am UTC](https://discuss.elastic.co/t/elasticsearch-deletes-documents-in-an-index-automatically/318316/13 "2022-11-07T11:37:30Z")
I also checked the logs when I was in prod. I didnt seem to find any log trace of documents being deleted ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 7, 2022, 11:39am UTC](https://discuss.elastic.co/t/elasticsearch-deletes-documents-in-an-index-automatically/318316/14 "2022-11-07T11:39:13Z")
That would not be logged unless you have audit logging. ---
**Author:** ![joanjanku2000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joanjanku2000/32/104542_2.png) [@joanjanku2000](https://discuss.elastic.co/u/joanjanku2000)\ **Post date:** [November 7, 2022, 11:39am UTC](https://discuss.elastic.co/t/elasticsearch-deletes-documents-in-an-index-automatically/318316/15 "2022-11-07T11:39:30Z")
I don't actually. ☹ ---
**Author:** ![joanjanku2000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joanjanku2000/32/104542_2.png) [@joanjanku2000](https://discuss.elastic.co/u/joanjanku2000)\ **Post date:** [November 7, 2022, 11:39am UTC](https://discuss.elastic.co/t/elasticsearch-deletes-documents-in-an-index-automatically/318316/16 "2022-11-07T11:39:46Z")
I should probably turn audit logging on ---
**Author:** ![joanjanku2000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joanjanku2000/32/104542_2.png) [@joanjanku2000](https://discuss.elastic.co/u/joanjanku2000)\ **Post date:** [November 7, 2022, 11:40am UTC](https://discuss.elastic.co/t/elasticsearch-deletes-documents-in-an-index-automatically/318316/17 "2022-11-07T11:40:06Z")
Would that tell me from where the deletion si comming though ? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 7, 2022, 11:41am UTC](https://discuss.elastic.co/t/elasticsearch-deletes-documents-in-an-index-automatically/318316/18 "2022-11-07T11:41:23Z")
It is not logged, so there is no way to tell. If you enable security, any script trying to delete without the correct credentials would fail. ---
**Author:** ![joanjanku2000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joanjanku2000/32/104542_2.png) [@joanjanku2000](https://discuss.elastic.co/u/joanjanku2000)\ **Post date:** [November 7, 2022, 11:45am UTC](https://discuss.elastic.co/t/elasticsearch-deletes-documents-in-an-index-automatically/318316/19 "2022-11-07T11:45:38Z")
But it would log the fact that something is trying to delete a specific document, right ? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 7, 2022, 11:51am UTC](https://discuss.elastic.co/t/elasticsearch-deletes-documents-in-an-index-automatically/318316/20 "2022-11-07T11:51:17Z")
No. Only audit logging would do that, and that I believe is a commercial feature. But if you secure the cluster and the deleting stops you might be able to conclude that something was deleting it. [Next page](https://discuss.elastic.co/t/elasticsearch-deletes-documents-in-an-index-automatically/318316.md?page=2)