# My index loses data every 30 days **URL:** \ **Category:** Elasticsearch\ **Created:** [October 2, 2019, 5:06pm UTC](https://discuss.elastic.co/t/my-index-loses-data-every-30-days/202015 "2019-10-02T17:06:40Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Lotfi\_Zaouai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lotfi_zaouai/32/46403_2.png) [@Lotfi\_Zaouai](https://discuss.elastic.co/u/Lotfi_Zaouai)\ **Post date:** [October 2, 2019, 5:06pm UTC](https://discuss.elastic.co/t/my-index-loses-data-every-30-days/202015/1 "2019-10-02T17:06:40Z")
## I use Elasticsearch and Kibana, I submit my data with the bulk API in [vb.net](http://vb.net) program. I did not make any special configuration in elastic search. here is the content of the file elasticsearch.yml : ## bootstrap.memory\_lock: false cluster.name: Mycluster http.port: 9200 network.host: 0.0.0.0.0.0 node.data: true node.ingest: true node.master: true node.max\_local\_storage\_nodes: 1 node.name: MyNode path.data: F:\Elasticsearch\data path.logs: F:\Elasticsearch\logs transport.tcp.port: 9300 xpack.license.self\_generated.type: basic xpack.security.enabled: false On average, I send 600 000 lines a day. In recent months I have noticed that every month I have a loss of data periodically instead of 600 miles lines on average, I have : 30-06-2019 : 188,626 31-07-2019 : 221,839 31-08-2019 : 206,808 30-09-2019 : 184,473 as shown in this screenshot ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/f/9f3620bd9cd9f0a6d80a1b412027d91b939eef89.png) Do you have an explanation for this phenomenon? Regards, Lotfi. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 2, 2019, 5:19pm UTC](https://discuss.elastic.co/t/my-index-loses-data-every-30-days/202015/2 "2019-10-02T17:19:01Z")
Do you analyse each bulk response to verify there were no errors and retry if there are? What version of Elasticsearch are you using? How large is your cluster and what does your Elasticsearch.yml file look like? ---
**Author:** ![Lotfi\_Zaouai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lotfi_zaouai/32/46403_2.png) [@Lotfi\_Zaouai](https://discuss.elastic.co/u/Lotfi_Zaouai)\ **Post date:** [October 2, 2019, 9:00pm UTC](https://discuss.elastic.co/t/my-index-loses-data-every-30-days/202015/3 "2019-10-02T21:00:49Z")
Hello Christian\_Dahlqvist, Thank you for your reply. 1- Yes I analyzed the bulk answers, no problem, the answer is like: [http://localhost:9200//\_bulk](http://localhost:9200//_bulk) {"took":380, "errors":false, 2- I use version 6.5.1 of Elasticsearch 3- Currently I have 242 GB of data 4- elasticsearch.yml : bootstrap.memory\_lock: false cluster.name: Mycluster http.port: 9200 network.host: 0.0.0.0.0.0 node.data: true node.ingest: true node.master: true node.max\_local\_storage\_nodes: 1 node.name: MyNode path.data: F:\Elasticsearch\data path.logs: F:\Elasticsearch\logs transport.tcp.port: 9300 xpack.license.self\_generated.type: basic xpack.security.enabled: false 5- I have a data loss every end of the month 30-06-2019 31-07-2019 31-08-2019 30-09-2019 ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/8/68a5e0f5f4547de6003f4d58506ffea2e3277829.png) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 3, 2019, 5:25am UTC](https://discuss.elastic.co/t/my-index-loses-data-every-30-days/202015/4 "2019-10-03T05:25:05Z")
What is the output of the [\_cluster/health API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html)? Are you using time-based indices, e.g. daily or monthly ones? Is there anything in the the Elasticsearch logs around the times you highlighted? Are you keeping statistics on how much you have indexed on the client side? ---
**Author:** ![Lotfi\_Zaouai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lotfi_zaouai/32/46403_2.png) [@Lotfi\_Zaouai](https://discuss.elastic.co/u/Lotfi_Zaouai)\ **Post date:** [October 3, 2019, 11:07am UTC](https://discuss.elastic.co/t/my-index-loses-data-every-30-days/202015/5 "2019-10-03T11:07:49Z")
1- { "cluster\_name" : "Mycluster", "status" : "yellow", "timed\_out" : false, "number\_of\_nodes" : 1, "number\_of\_data\_nodes" : 1, "active\_primary\_shards" : 17, "active\_shards" : 17, "relocating\_shards" : 0, "initializing\_shards" : 0, "unassigned\_shards" : 15, "delayed\_unassigned\_shards" : 0, "number\_of\_pending\_tasks" : 0, "number\_of\_in\_flight\_fetch" : 0, "task\_max\_waiting\_in\_queue\_millis" : 0, "active\_shards\_percent\_as\_number" : 53.125 } 2- no time-based index is used, on the contrary, I keep my data for 3 years to make annual statistics 3- MyCluster-2019-09-30.log : the majority of the log file contains lines of this kind ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/3/b348c4308eee8d894c40f2fbca7e073c77581cbe.png) 4- Yes, as I have already said, my client program sends, on average 600 thousand lines per day, even in the days when there is a problem he sent the data with the same frequency . ---
**Author:** ![Lotfi\_Zaouai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lotfi_zaouai/32/46403_2.png) [@Lotfi\_Zaouai](https://discuss.elastic.co/u/Lotfi_Zaouai)\ **Post date:** [October 8, 2019, 7:26am UTC](https://discuss.elastic.co/t/my-index-loses-data-every-30-days/202015/6 "2019-10-08T07:26:30Z")
Helo, Any kind of help will be appreciable. Regards, Lotfi. ---
**Author:** ![ywelsch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ywelsch/32/7751_2.png) [@ywelsch](https://discuss.elastic.co/u/ywelsch)\ **Post date:** [October 9, 2019, 9:12am UTC](https://discuss.elastic.co/t/my-index-loses-data-every-30-days/202015/7 "2019-10-09T09:12:05Z")
Are you checking the responses for your bulk requests in your application? The cluster might be rejecting requests due to load, or fail certain requests, which means that you will have to retry those. In particular are you checking whether `errors : false` in the bulk response? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 6, 2019, 9:12am UTC](https://discuss.elastic.co/t/my-index-loses-data-every-30-days/202015/8 "2019-11-06T09:12:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.