# Data disappearing and new uuid **URL:** \ **Category:** Elasticsearch\ **Created:** [April 6, 2019, 4:03pm UTC](https://discuss.elastic.co/t/data-disappearing-and-new-uuid/175678 "2019-04-06T16:03:17Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![Callonski](https://avatars.discourse-cdn.com/v4/letter/c/46a35a/32.png) [@Callonski](https://discuss.elastic.co/u/Callonski)\ **Post date:** [April 6, 2019, 4:03pm UTC](https://discuss.elastic.co/t/data-disappearing-and-new-uuid/175678/1 "2019-04-06T16:03:18Z")
I'm trying to load around 20 GB of data into an index. My problem is that while I do this, all of a sudden the uuid, docs.count and store.size changes after querying: curl [http://localhost:9200/\_cat/indices?v](http://localhost:9200/_cat/indices?v) An example: green open be eEufIf0GTUmBNsFOhgu7Nw 5 1 935877 41632 15.8gb 7.9gb I get this response. Then after some more files have been uploaded into ES, same query gives me: green open be oZxXwAAoRiS0Vt6XsGSUzw 5 1 462018 7922 4.6gb 2.2gb A different uuid, half the number of docs and a third of the storage. Anyone has a clue of what's happening? My cluster is totally isolated from internet within GCP with no external ip:s. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [April 6, 2019, 4:34pm UTC](https://discuss.elastic.co/t/data-disappearing-and-new-uuid/175678/2 "2019-04-06T16:34:37Z")
Could you share your logs? Formatted please. ---
**Author:** ![Callonski](https://avatars.discourse-cdn.com/v4/letter/c/46a35a/32.png) [@Callonski](https://discuss.elastic.co/u/Callonski)\ **Post date:** [April 6, 2019, 5:44pm UTC](https://discuss.elastic.co/t/data-disappearing-and-new-uuid/175678/3 "2019-04-06T17:44:21Z")
``` 2019-04-06T15:35:06,021][WARN][o.e.a.b.TransportShardBulkAction] [elasticsearch-elastic-vm-0] unexpected error during the primary phase for action [indices:data/write/bulk[s]], request [BulkShardRequest [[be][4]] containing [524] requests] org.elasticsearch.index.IndexNotFoundException: no such index at org.elasticsearch.cluster.routing.RoutingTable.shardRoutingTable(RoutingTable.java:137) ~ . [elasticsearch-5.6.8.jar:5.6.8] at org.elasticsearch.action.support.replication.TransportReplicationAction $ReroutePhase.primary(TransportReplicationAction.java:745) ~[elasticsearch-5.6.8.jar:5.6.8] at org.elasticsearch.action.support.replication.TransportReplicationAction $ReroutePhase.doRun(TransportReplicationAction.java:680) ~[elasticsearch-5.6.8.jar:5.6.8] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~ [elasticsearch-5.6.8.jar:5.6.8] at org.elasticsearch.action.support.replication.TransportReplicationAction $ReroutePhase$2.onNewClusterState(TransportReplicationAction.java:835) ~[elasticsearch- 5.6.8.jar:5.6.8] at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener. onNewClusterState(ClusterStateObserver.java:297) ~[elasticsearch-5.6.8.jar:5.6.8] at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener. clusterChanged(ClusterStateObserver.java:185) ~[elasticsearch-5.6.8.jar:5.6.8] at org.elasticsearch.cluster.service.ClusterService.lambda$publishAndApplyChanges $7(ClusterService.java:777) ~[elasticsearch-5.6.8.jar:5.6.8] at java.util.concurrent.ConcurrentHashMap$KeySpliterator.forEachRemaining (ConcurrentHashMap.java:3527) [?:1.8.0_181] at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:743) [?:1.8.0_181] at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) [?:1.8.0_181] at org.elasticsearch.cluster.service.ClusterService.publishAndApplyChanges (ClusterService.java:774) [elasticsearch-5.6.8.jar:5.6.8] at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:587) [elasticsearch-5.6.8.jar:5.6.8] at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher. run(ClusterService.java:263) [elasticsearch-5.6.8.jar:5.6.8] at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-5.6.8.jar:5.6.8] at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-5.6.8.jar:5.6.8] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run (ThreadContext.java:575) [elasticsearch-5.6.8.jar:5.6.8] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor $TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [elasticsearch-5.6.8.jar:5.6.8] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor $TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [elasticsearch- 5.6.8.jar:5.6.8] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181] [2019-04-06T15:35:06,028][WARN][o.e.a.b.TransportShardBulkAction] [elasticsearch-elastic-vm-0] unexpected error during the primary phase for action [indices:data/write/bulk[s]], request [BulkShardRequest [[be][1]] containing [468] requests] org.elasticsearch.index.IndexNotFoundException: no such index at org.elasticsearch.cluster.routing.RoutingTable.shardRoutingTable(RoutingTable.java:137) ~ . [elasticsearch-5.6.8.jar:5.6.8] at org.elasticsearch.action.support.replication.TransportReplicationAction $ReroutePhase.primary(TransportReplicationAction.java:745) ~[elasticsearch-5.6.8.jar:5.6.8] at org.elasticsearch.action.support.replication.TransportReplicationAction $ReroutePhase.doRun(TransportReplicationAction.java:680) ~[elasticsearch-5.6.8.jar:5.6.8] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~ [elasticsearch-5.6.8.jar:5.6.8] at org.elasticsearch.action.support.replication.TransportReplicationAction $ReroutePhase$2.onNewClusterState(TransportReplicationAction.java:835) ~[elasticsearch- 5.6.8.jar:5.6.8] (TransportReplicationAction.java:835) ~[elasticsearch-5.6.8.jar:5.6.8] at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener. onNewClusterState(ClusterStateObserver.java:297) ~[elasticsearch-5.6.8.jar:5.6.8] at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener. clusterChanged(ClusterStateObserver.java:185) ~[elasticsearch-5.6.8.jar:5.6.8] at org.elasticsearch.cluster.service.ClusterService.lambda$publishAndApplyChanges $7(ClusterService.java:777) ~[elasticsearch-5.6.8.jar:5.6.8] at java.util.concurrent.ConcurrentHashMap$KeySpliterator.forEachRemaining (ConcurrentHashMap.java:3527) [?:1.8.0_181] at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:743) [?:1.8.0_181] at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) [?:1.8.0_181] at org.elasticsearch.cluster.service.ClusterService.publishAndApplyChanges(ClusterService.java:774) [elasticsearch-5.6.8.jar:5.6.8] at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:587) [elasticsearch-5.6.8.jar:5.6.8] at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) [elasticsearch-5.6.8.jar:5.6.8] at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-5.6.8.jar:5.6.8] at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-5.6.8.jar:5.6.8] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.8.jar:5.6.8] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [elasticsearch-5.6.8.jar:5.6.8] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [elasticsearch-5.6.8.jar:5.6.8] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181] ``` ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [April 6, 2019, 8:21pm UTC](https://discuss.elastic.co/t/data-disappearing-and-new-uuid/175678/4 "2019-04-06T20:21:24Z")
Could you share the full logs? You can upload them on [gist.github.com](http://gist.github.com) ---
**Author:** ![Callonski](https://avatars.discourse-cdn.com/v4/letter/c/46a35a/32.png) [@Callonski](https://discuss.elastic.co/u/Callonski)\ **Post date:** [April 7, 2019, 10:03am UTC](https://discuss.elastic.co/t/data-disappearing-and-new-uuid/175678/5 "2019-04-07T10:03:27Z")
> > > > > > Pew, thats a lot of logs 🙂 I hope I got all the relevant ones. I'm quite new to elasticsearch, but it seems to be some issue with the bulk-import, that it can't find the index and then creates a new one? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 7, 2019, 10:14am UTC](https://discuss.elastic.co/t/data-disappearing-and-new-uuid/175678/6 "2019-04-07T10:14:29Z")
That is a lot of logs, but not many of them come from Elasticsearch. Could you share more of the Elasticsearch server logs? Also can you tell us a bit more about this cluster? How many nodes does it have? Can you share the `elasticsearch.yml` files from each node? ---
**Author:** ![Callonski](https://avatars.discourse-cdn.com/v4/letter/c/46a35a/32.png) [@Callonski](https://discuss.elastic.co/u/Callonski)\ **Post date:** [April 7, 2019, 11:00am UTC](https://discuss.elastic.co/t/data-disappearing-and-new-uuid/175678/7 "2019-04-07T11:00:46Z")
My cluster has 3 nodes and elasticsearch.yml looks like this: (the other looks the same, with different names) ``` cluster.name: "elasticsearch-cluster" node.name: "elasticsearch-elastic-vm-0" path.data: /var/lib/elasticsearch/data network.host: 0.0.0.0 discovery.zen.ping_timeout: 30s discovery.zen.fd.ping_timeout: 30s discovery.zen.minimum_master_nodes: 2 discovery.zen.ping.unicast.hosts: [ "elasticsearch-elastic-vm-0", "elasticsearch-elastic-vm-1", "elasticsearch-elastic-vm-2" ] ``` The server logs, is it these? ``` elasticsearch-cluster-2019-04-02.log elasticsearch-cluster-2019-04-05.log elasticsearch- cluster_index_search_slowlog.log elasticsearch-cluster-2019-04-03.log elasticsearch-cluster_deprecation.log elasticsearch-cluster.log elasticsearch-cluster-2019-04-04.log elasticsearch-cluster_index_indexing_slowlog.log ``` > ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 7, 2019, 11:39am UTC](https://discuss.elastic.co/t/data-disappearing-and-new-uuid/175678/8 "2019-04-07T11:39:04Z")
> [@Callonski](#): > > The server logs, is it these? Yes, particularly `elasticsearch-cluster.log` and the ones of the form `elasticsearch-cluster-YYYY-MM-DD.log`. Are you saying that they _only_ contain this `unexpected error during the primary phase`, on all three of your nodes? ---
**Author:** ![Callonski](https://avatars.discourse-cdn.com/v4/letter/c/46a35a/32.png) [@Callonski](https://discuss.elastic.co/u/Callonski)\ **Post date:** [April 7, 2019, 12:07pm UTC](https://discuss.elastic.co/t/data-disappearing-and-new-uuid/175678/9 "2019-04-07T12:07:57Z")
Ok, here's logs from all three nodes. > > > > > > Node 0, didn't log anything yesterday 2019-04-06. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 7, 2019, 12:20pm UTC](https://discuss.elastic.co/t/data-disappearing-and-new-uuid/175678/10 "2019-04-07T12:20:08Z")
Thanks. The key line is this one: ```auto [2019-04-07T09:12:15,357][INFO][o.e.c.m.MetaDataDeleteIndexService] [elasticsearch-elastic-vm-1] [be/Mae2qxonSxKWCPgJwL6Wlg] deleting index ``` Someone or something is deleting the index. ---
**Author:** ![Callonski](https://avatars.discourse-cdn.com/v4/letter/c/46a35a/32.png) [@Callonski](https://discuss.elastic.co/u/Callonski)\ **Post date:** [April 7, 2019, 12:37pm UTC](https://discuss.elastic.co/t/data-disappearing-and-new-uuid/175678/11 "2019-04-07T12:37:07Z")
Thanks for the help! I'm the only one with access to the cluster, so what could trigger a delete of an Index like that? Am I loading too much data too fast? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 7, 2019, 12:39pm UTC](https://discuss.elastic.co/t/data-disappearing-and-new-uuid/175678/12 "2019-04-07T12:39:36Z")
Is your cluster open to the internet and accessible via a public IP? ---
**Author:** ![Callonski](https://avatars.discourse-cdn.com/v4/letter/c/46a35a/32.png) [@Callonski](https://discuss.elastic.co/u/Callonski)\ **Post date:** [April 7, 2019, 12:41pm UTC](https://discuss.elastic.co/t/data-disappearing-and-new-uuid/175678/13 "2019-04-07T12:41:49Z")
Nope, no external IP:s. It's on GCP within the VPC(virtual private network) and I access it directly from Appengine. If I need to get into the machines, I spin up a bastion-host to SSH into the machine with ES. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 7, 2019, 1:22pm UTC](https://discuss.elastic.co/t/data-disappearing-and-new-uuid/175678/14 "2019-04-07T13:22:22Z")
> [@Callonski](#): > > what could trigger a delete of an Index like that? Am I loading too much data too fast? No, Elasticsearch definitely doesn't delete anything unless explicitly instructed. From Elasticsearch's point of view there must be something calling `DELETE /be` (or the transport client equivalent) or using the `remove_index` command of the [Index Aliases API](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/indices-aliases.html). ---
**Author:** ![Callonski](https://avatars.discourse-cdn.com/v4/letter/c/46a35a/32.png) [@Callonski](https://discuss.elastic.co/u/Callonski)\ **Post date:** [April 8, 2019, 8:58am UTC](https://discuss.elastic.co/t/data-disappearing-and-new-uuid/175678/15 "2019-04-08T08:58:31Z")
There was a bug in my code 😊. I thought I was deleting a document but ended up deleting the entire index. My mistake, Thanks for the help! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 6, 2019, 8:58am UTC](https://discuss.elastic.co/t/data-disappearing-and-new-uuid/175678/16 "2019-05-06T08:58:34Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.