# Lost data in ElasticSearch cluster after disconnected node **URL:** \ **Category:** Elasticsearch\ **Created:** [March 25, 2013, 3:45pm UTC](https://discuss.elastic.co/t/lost-data-in-elasticsearch-cluster-after-disconnected-node/11296 "2013-03-25T15:45:03Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![iveta\_ruda](https://avatars.discourse-cdn.com/v4/letter/i/b3f665/32.png) [@iveta\_ruda](https://discuss.elastic.co/u/iveta_ruda)\ **Post date:** [March 25, 2013, 3:45pm UTC](https://discuss.elastic.co/t/lost-data-in-elasticsearch-cluster-after-disconnected-node/11296/1 "2013-03-25T15:45:03Z")
I was simulating a split-brain cluster in a 3 node cluster. I had set minimum master nodes to 2, number of replicas to 2. ( I did not change node.master nor node.data settings.) There were created some indexes with some data, and status of the cluster was green. Then I simulated that one node was disconnected, i.e. I blocked a tcp port for this node. Then I added some data to one of the left nodes ( the status was yellow ). The data were there, but after unblocking the port for the node, the status of the cluster was green but the added data diappeared from the whole cluster. I was checking it on ES version 0.19.10. ( I can check it on a new version 0.20.3 we have, to be sure, but I doubt it will be correct). We are using GUI to save data to ORACLE and only some data are sent to ES cluster via REST API, and it is unwanted state when in case of a connection failure the data integrity will be broken. Is there is a way how to avoid this? Thanks. Iveta -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [March 25, 2013, 7:11pm UTC](https://discuss.elastic.co/t/lost-data-in-elasticsearch-cluster-after-disconnected-node/11296/2 "2013-03-25T19:11:05Z")
On Mon, 2013-03-25 at 08:45 -0700, [iveta.ruda@performgroup.com](mailto:iveta.ruda@performgroup.com) wrote: > I was simulating a split-brain cluster in a 3 node cluster. I had set > minimum master nodes to 2, number of replicas to 2. ( I did not change > node.master nor node.data settings.) There were created some indexes > with some data, and status of the cluster was green. Then I simulated > that one node was disconnected, i.e. I blocked a tcp port for this > node. Then I added some data to one of the left nodes ( the status was > yellow ). The data were there, but after unblocking the port for the > node, the status of the cluster was green but the added data > diappeared from the whole cluster. > I was checking it on ES version 0.19.10. ( I can check it on a new > version 0.20.3 we have, to be sure, but I doubt it will be correct). > We are using GUI to save data to ORACLE and only some data are sent to > ES cluster via REST API, and it is unwanted state when in case of a > connection failure the data integrity will be broken. > Is there is a way how to avoid this? This should work. Can you post a full recreation? thanks Clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![iveta\_ruda](https://avatars.discourse-cdn.com/v4/letter/i/b3f665/32.png) [@iveta\_ruda](https://discuss.elastic.co/u/iveta_ruda)\ **Post date:** [March 26, 2013, 11:11am UTC](https://discuss.elastic.co/t/lost-data-in-elasticsearch-cluster-after-disconnected-node/11296/3 "2013-03-26T11:11:38Z")
I have enclosed elasticsearch.yml file of one node from the cluster. The others are similar. It is just the testing cluster, there are some indexes created, but not a lot of data. The process I simulated is as follows. The status of the 3 node cluster was green. I checked it via Elasticsearch head and checked data there. Everything was OK. Then I blocked port 9300 of one node via: iptables -A INPUT -p tcp --destination-port 9300 -j DROP Then I checked the status of the cluster. It was yellow and the blocked node was disconnected from the cluster. Then I added some document to one index via PUT request, then I checked, whether there were added data via ES head and ( e.g. [http://31.222.171.66:8080/livestream/\_search?pretty&size=1000](http://31.222.171.66:8080/livestream/_search?pretty&size=1000) ) from both nodes still working in the cluster. The added data were there. Then I unblocked port 9300 of the blocked node via: iptables -D INPUT 1 ( and checked via iptables --list ) Then the status of the cluster was green, all the nodes were working OK. But when I checked data for the index, the previously added data dissappeared from all nodes. On Monday, 25 March 2013 20:11:05 UTC+1, Clinton Gormley wrote: > On Mon, 2013-03-25 at 08:45 -0700, [iveta...@performgroup.com](mailto:iveta...@performgroup.com) \wrote: > > > I was simulating a split-brain cluster in a 3 node cluster. I had set > > minimum master nodes to 2, number of replicas to 2. ( I did not change > > node.master nor node.data settings.) There were created some indexes > > with some data, and status of the cluster was green. Then I simulated > > that one node was disconnected, i.e. I blocked a tcp port for this > > node. Then I added some data to one of the left nodes ( the status was > > yellow ). The data were there, but after unblocking the port for the > > node, the status of the cluster was green but the added data > > diappeared from the whole cluster. > > I was checking it on ES version 0.19.10. ( I can check it on a new > > version 0.20.3 we have, to be sure, but I doubt it will be correct). > > We are using GUI to save data to ORACLE and only some data are sent to > > ES cluster via REST API, and it is unwanted state when in case of a > > connection failure the data integrity will be broken. > > Is there is a way how to avoid this? > > This should work. Can you post a full recreation? > > thanks > > Clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![iveta\_ruda](https://avatars.discourse-cdn.com/v4/letter/i/b3f665/32.png) [@iveta\_ruda](https://discuss.elastic.co/u/iveta_ruda)\ **Post date:** [March 26, 2013, 11:14am UTC](https://discuss.elastic.co/t/lost-data-in-elasticsearch-cluster-after-disconnected-node/11296/4 "2013-03-26T11:14:14Z")
I have enclosed elasticsearch.yml file of one node from the cluster. The others are similar. It is just the testing cluster, there are some indexes created, but not a lot of data. The process I simulated is as follows. The status of the 3 node cluster was green. I checked it via Elasticsearch head and checked data there. Everything was OK. Then I blocked port 9300 of one node via: iptables -A INPUT -p tcp --destination-port 9300 -j DROP Then I checked the status of the cluster. It was yellow and the blocked node was disconnected from the cluster. Then I added some document to one index via PUT request, then I checked, whether there were added data via ES head and ( e.g. [http://31.222.171.66:8080/livestream/\_search?pretty&size=1000](http://31.222.171.66:8080/livestream/_search?pretty&size=1000) ) from both nodes still working in the cluster. The added data were there. Then I unblocked port 9300 of the blocked node via: iptables -D INPUT 1 ( and checked via iptables --list ) Then the status of the cluster was green, all the nodes were working OK. But when I checked data for the index, the previously added data dissappeared from all nodes. On Monday, 25 March 2013 20:11:05 UTC+1, Clinton Gormley wrote: > On Mon, 2013-03-25 at 08:45 -0700, [iveta...@performgroup.com](mailto:iveta...@performgroup.com) \wrote: > > > I was simulating a split-brain cluster in a 3 node cluster. I had set > > minimum master nodes to 2, number of replicas to 2. ( I did not change > > node.master nor node.data settings.) There were created some indexes > > with some data, and status of the cluster was green. Then I simulated > > that one node was disconnected, i.e. I blocked a tcp port for this > > node. Then I added some data to one of the left nodes ( the status was > > yellow ). The data were there, but after unblocking the port for the > > node, the status of the cluster was green but the added data > > diappeared from the whole cluster. > > I was checking it on ES version 0.19.10. ( I can check it on a new > > version 0.20.3 we have, to be sure, but I doubt it will be correct). > > We are using GUI to save data to ORACLE and only some data are sent to > > ES cluster via REST API, and it is unwanted state when in case of a > > connection failure the data integrity will be broken. > > Is there is a way how to avoid this? > > This should work. Can you post a full recreation? > > thanks > > Clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![iveta\_ruda](https://avatars.discourse-cdn.com/v4/letter/i/b3f665/32.png) [@iveta\_ruda](https://discuss.elastic.co/u/iveta_ruda)\ **Post date:** [March 26, 2013, 11:18am UTC](https://discuss.elastic.co/t/lost-data-in-elasticsearch-cluster-after-disconnected-node/11296/5 "2013-03-26T11:18:19Z")
I have had a problem enclosing attachement, so I am trying once again with renaming it to elasticsearch.txt. On Tuesday, 26 March 2013 12:14:14 UTC+1, [iveta...@performgroup.com](mailto:iveta...@performgroup.com) wrote: > I have enclosed elasticsearch.yml file of one node from the cluster. The > others are similar. It is just the testing cluster, there are some indexes > created, but not a lot of data. > The process I simulated is as follows. > The status of the 3 node cluster was green. I checked it via Elasticsearch > head and checked data there. Everything was OK. Then I blocked port 9300 of > one node via: > iptables -A INPUT -p tcp --destination-port 9300 -j DROP > Then I checked the status of the cluster. It was yellow and the blocked > node was disconnected from the cluster. Then I added some document to one > index via PUT request, > then I checked, whether there were added data via ES head and ( e.g. > [http://31.222.171.66:8080/livestream/\_search?pretty&size=1000](http://31.222.171.66:8080/livestream/_search?pretty&size=1000) ) from both > nodes still working in the > cluster. The added data were there. > Then I unblocked port 9300 of the blocked node via: > iptables -D INPUT 1 ( and checked via iptables --list ) > Then the status of the cluster was green, all the nodes were working OK. > But when I checked data for the index, the previously added data > dissappeared from all nodes. > > On Monday, 25 March 2013 20:11:05 UTC+1, Clinton Gormley wrote: > > > On Mon, 2013-03-25 at 08:45 -0700, [iveta...@performgroup.com](mailto:iveta...@performgroup.com) wrote: > > > > > I was simulating a split-brain cluster in a 3 node cluster. I had set > > > minimum master nodes to 2, number of replicas to 2. ( I did not change > > > node.master nor node.data settings.) There were created some indexes > > > with some data, and status of the cluster was green. Then I simulated > > > that one node was disconnected, i.e. I blocked a tcp port for this > > > node. Then I added some data to one of the left nodes ( the status was > > > yellow ). The data were there, but after unblocking the port for the > > > node, the status of the cluster was green but the added data > > > diappeared from the whole cluster. > > > I was checking it on ES version 0.19.10. ( I can check it on a new > > > version 0.20.3 we have, to be sure, but I doubt it will be correct). > > > We are using GUI to save data to ORACLE and only some data are sent to > > > ES cluster via REST API, and it is unwanted state when in case of a > > > connection failure the data integrity will be broken. > > > Is there is a way how to avoid this? > > > > This should work. Can you post a full recreation? > > > > thanks > > > > Clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![iveta\_ruda](https://avatars.discourse-cdn.com/v4/letter/i/b3f665/32.png) [@iveta\_ruda](https://discuss.elastic.co/u/iveta_ruda)\ **Post date:** [March 26, 2013, 12:23pm UTC](https://discuss.elastic.co/t/lost-data-in-elasticsearch-cluster-after-disconnected-node/11296/6 "2013-03-26T12:23:18Z")
I am sorry, it was my fault. I have sent PUT request to ES via our application, but there was set small TTL value and therefore the data expired quickly, so that's why they were thew away. No need to bother any more. On Tuesday, 26 March 2013 12:18:19 UTC+1, [iveta...@performgroup.com](mailto:iveta...@performgroup.com) wrote: > I have had a problem enclosing attachement, so I am trying once again with > renaming it to elasticsearch.txt. > > On Tuesday, 26 March 2013 12:14:14 UTC+1, [iveta...@performgroup.com](mailto:iveta...@performgroup.com) wrote: > > > I have enclosed elasticsearch.yml file of one node from the cluster. The > > others are similar. It is just the testing cluster, there are some indexes > > created, but not a lot of data. > > The process I simulated is as follows. > > The status of the 3 node cluster was green. I checked it via > > Elasticsearch head and checked data there. Everything was OK. Then I > > blocked port 9300 of one node via: > > iptables -A INPUT -p tcp --destination-port 9300 -j DROP > > Then I checked the status of the cluster. It was yellow and the blocked > > node was disconnected from the cluster. Then I added some document to one > > index via PUT request, > > then I checked, whether there were added data via ES head and ( e.g. > > [http://31.222.171.66:8080/livestream/\_search?pretty&size=1000](http://31.222.171.66:8080/livestream/_search?pretty&size=1000) ) from > > both nodes still working in the > > cluster. The added data were there. > > Then I unblocked port 9300 of the blocked node via: > > iptables -D INPUT 1 ( and checked via iptables --list ) > > Then the status of the cluster was green, all the nodes were working OK. > > But when I checked data for the index, the previously added data > > dissappeared from all nodes. > > > > On Monday, 25 March 2013 20:11:05 UTC+1, Clinton Gormley wrote: > > > > > On Mon, 2013-03-25 at 08:45 -0700, [iveta...@performgroup.com](mailto:iveta...@performgroup.com) wrote: > > > > > > > I was simulating a split-brain cluster in a 3 node cluster. I had set > > > > minimum master nodes to 2, number of replicas to 2. ( I did not change > > > > node.master nor node.data settings.) There were created some indexes > > > > with some data, and status of the cluster was green. Then I simulated > > > > that one node was disconnected, i.e. I blocked a tcp port for this > > > > node. Then I added some data to one of the left nodes ( the status was > > > > yellow ). The data were there, but after unblocking the port for the > > > > node, the status of the cluster was green but the added data > > > > diappeared from the whole cluster. > > > > I was checking it on ES version 0.19.10. ( I can check it on a new > > > > version 0.20.3 we have, to be sure, but I doubt it will be correct). > > > > We are using GUI to save data to ORACLE and only some data are sent to > > > > ES cluster via REST API, and it is unwanted state when in case of a > > > > connection failure the data integrity will be broken. > > > > Is there is a way how to avoid this? > > > > > > This should work. Can you post a full recreation? > > > > > > thanks > > > > > > Clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:44am UTC](https://discuss.elastic.co/t/lost-data-in-elasticsearch-cluster-after-disconnected-node/11296/7 "2017-07-06T02:44:25Z")