# Loosing data in elasticsearch **URL:** \ **Category:** Elasticsearch\ **Created:** [July 10, 2017, 9:07am UTC](https://discuss.elastic.co/t/loosing-data-in-elasticsearch/92433 "2017-07-10T09:07:38Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Idarlington](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/idarlington/32/19887_2.png) [@Idarlington](https://discuss.elastic.co/u/Idarlington)\ **Post date:** [July 10, 2017, 9:07am UTC](https://discuss.elastic.co/t/loosing-data-in-elasticsearch/92433/1 "2017-07-10T09:07:38Z")
I have elasticsearch 2.0.0 set up with kafka elasticsearch connector. I recently discovered I am loosing data. I can't see some old data anymore. Data is lost in a FIFO way. At first, I thought it was a heap error and increased my heap size but it is still happening. Has anyone experienced this? Any pointers on what to do? I Would appreciate all help. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 10, 2017, 9:10am UTC](https://discuss.elastic.co/t/loosing-data-in-elasticsearch/92433/2 "2017-07-10T09:10:09Z")
How do you know it is being lost? ---
**Author:** ![Idarlington](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/idarlington/32/19887_2.png) [@Idarlington](https://discuss.elastic.co/u/Idarlington)\ **Post date:** [July 10, 2017, 9:11am UTC](https://discuss.elastic.co/t/loosing-data-in-elasticsearch/92433/3 "2017-07-10T09:11:36Z")
I search for the data based on the period and can't find it. I also have a graphical display of the data on a webpage so that was what alerted me to this ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 10, 2017, 9:42am UTC](https://discuss.elastic.co/t/loosing-data-in-elasticsearch/92433/4 "2017-07-10T09:42:33Z")
And are you sure it's making it to kafka and to Elasticsearch? ---
**Author:** ![Idarlington](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/idarlington/32/19887_2.png) [@Idarlington](https://discuss.elastic.co/u/Idarlington)\ **Post date:** [July 10, 2017, 9:44am UTC](https://discuss.elastic.co/t/loosing-data-in-elasticsearch/92433/5 "2017-07-10T09:44:37Z")
Yes, the data is already saved and I could see it but after a while it is gone from elasticsearch ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 10, 2017, 9:47am UTC](https://discuss.elastic.co/t/loosing-data-in-elasticsearch/92433/6 "2017-07-10T09:47:17Z")
Have you by any chance got Curator setup in a cron job to delete old indices? Is there anything in the Elasticsearch logs about indices being deleted? ---
**Author:** ![Idarlington](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/idarlington/32/19887_2.png) [@Idarlington](https://discuss.elastic.co/u/Idarlington)\ **Post date:** [July 10, 2017, 9:49am UTC](https://discuss.elastic.co/t/loosing-data-in-elasticsearch/92433/7 "2017-07-10T09:49:49Z")
No this is a new setup, I have not set up curator yet. I checked logs but couldn't see anything, I could share my logs if that helps I installed elasticsearch from the source (apt-get) ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 10, 2017, 9:51am UTC](https://discuss.elastic.co/t/loosing-data-in-elasticsearch/92433/8 "2017-07-10T09:51:31Z")
> [@Idarlington](#): > > Yes, the data is already saved and I could see it but after a while it is gone from elasticsearch Can you explain this more please. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 10, 2017, 9:53am UTC](https://discuss.elastic.co/t/loosing-data-in-elasticsearch/92433/9 "2017-07-10T09:53:29Z")
How did you verify the data reached Elasticsearch? ---
**Author:** ![Idarlington](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/idarlington/32/19887_2.png) [@Idarlington](https://discuss.elastic.co/u/Idarlington)\ **Post date:** [July 10, 2017, 10:24am UTC](https://discuss.elastic.co/t/loosing-data-in-elasticsearch/92433/10 "2017-07-10T10:24:09Z")
This: > [@Idarlington](#): > > I search for the data based on the period and can't find it. I also have a graphical display of the data on a webpage so that was what alerted me to this ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 10, 2017, 10:26am UTC](https://discuss.elastic.co/t/loosing-data-in-elasticsearch/92433/11 "2017-07-10T10:26:17Z")
Does this mean that you inspected the data through Kibana? Did you query Elasticsearch through the APIs? ---
**Author:** ![Idarlington](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/idarlington/32/19887_2.png) [@Idarlington](https://discuss.elastic.co/u/Idarlington)\ **Post date:** [July 10, 2017, 10:28am UTC](https://discuss.elastic.co/t/loosing-data-in-elasticsearch/92433/12 "2017-07-10T10:28:32Z")
So I am ingesting data from kafka into elasticsearch using the [kafka elasticsearch connector](https://github.com/confluentinc/kafka-connect-elasticsearch) I can view the data as it comes in via a graph (I have a graph aggregated by day created with d3), after a while I notice days that formerly had data are empty. So I check using sense and I find that all data or most times some part of that data is missing. This is data I verified formerly was there. Hope this helps? ---
**Author:** ![Idarlington](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/idarlington/32/19887_2.png) [@Idarlington](https://discuss.elastic.co/u/Idarlington)\ **Post date:** [July 10, 2017, 10:30am UTC](https://discuss.elastic.co/t/loosing-data-in-elasticsearch/92433/13 "2017-07-10T10:30:10Z")
I used sense to search within the period that was missing ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 10, 2017, 10:31am UTC](https://discuss.elastic.co/t/loosing-data-in-elasticsearch/92433/14 "2017-07-10T10:31:42Z")
Elasticsearch won't just delete the data, so something else must be requesting it. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 10, 2017, 10:33am UTC](https://discuss.elastic.co/t/loosing-data-in-elasticsearch/92433/15 "2017-07-10T10:33:27Z")
Are you using time-based indices? What does `GET /_cat/indices` show? ---
**Author:** ![Idarlington](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/idarlington/32/19887_2.png) [@Idarlington](https://discuss.elastic.co/u/Idarlington)\ **Post date:** [July 10, 2017, 10:35am UTC](https://discuss.elastic.co/t/loosing-data-in-elasticsearch/92433/16 "2017-07-10T10:35:32Z")
> [@Christian\_Dahlqvist](#): > > GET /\_cat/indices ```auto yellow open grits 5 1 23408051 4208025 2.7gb 2.7gb yellow open .kibana 1 1 1 0 2.9kb 2.9kb yellow open test-elasticsearch-sink 5 1 0 0 785b 785b yellow open grit 5 1 0 0 785b 785b ``` grits is the main index ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 10, 2017, 10:41am UTC](https://discuss.elastic.co/t/loosing-data-in-elasticsearch/92433/17 "2017-07-10T10:41:17Z")
Are you assigning an external ID to the documents you are indexing or are you allowing Elasticsearch to automatically generate an ID? ---
**Author:** ![Idarlington](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/idarlington/32/19887_2.png) [@Idarlington](https://discuss.elastic.co/u/Idarlington)\ **Post date:** [July 10, 2017, 10:44am UTC](https://discuss.elastic.co/t/loosing-data-in-elasticsearch/92433/18 "2017-07-10T10:44:14Z")
Kafka elasticsearch connector generates ids : [http://docs.confluent.io/current/connect/connect-elasticsearch/docs/elasticsearch\_connector.html](http://docs.confluent.io/current/connect/connect-elasticsearch/docs/elasticsearch_connector.html) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 10, 2017, 10:50am UTC](https://discuss.elastic.co/t/loosing-data-in-elasticsearch/92433/19 "2017-07-10T10:50:28Z")
The index stats shows that you have deleted documents in your index. This may indicate that you have indexed multiple documents with the same ID, which would cause an update (delete + creation of new document) or that you have deleted documents by some other means, e.g. through the delete-by-query API. Elasticsearch does not by itself delete data. If you have the ID of a document that you would expect to find in the period for which data is no longer found you could look this up and see if it has been updated or deleted. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 10, 2017, 11:09am UTC](https://discuss.elastic.co/t/loosing-data-in-elasticsearch/92433/20 "2017-07-10T11:09:55Z")
As you are using such an old version of Elasticsearch, it just struck me that it might be possible that TTL is enabled, which would cause documents to get deleted. It was listed as deprecated in 2.0, but may still have been available. Get the settings for the index through the [get index settings API](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/indices-get-settings.html) and also check the [mappings for the index](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/indices-get-mapping.html) and look for any `ttl` related settings. [Next page](https://discuss.elastic.co/t/loosing-data-in-elasticsearch/92433.md?page=2)