# Data loss on shutdown of node in a 3 node cluster

**URL:** <https://discuss.elastic.co/t/data-loss-on-shutdown-of-node-in-a-3-node-cluster/13977>\
**Category:** Elasticsearch\
**Created:** [October 16, 2013, 8:34pm UTC](https://discuss.elastic.co/t/data-loss-on-shutdown-of-node-in-a-3-node-cluster/13977 "2013-10-16T20:34:01Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Gregory\_Durham](https://avatars.discourse-cdn.com/v4/letter/g/3ec8ea/32.png) [@Gregory\_Durham](https://discuss.elastic.co/u/Gregory_Durham)\
**Post date:** [October 16, 2013, 8:34pm UTC](https://discuss.elastic.co/t/data-loss-on-shutdown-of-node-in-a-3-node-cluster/13977/1 "2013-10-16T20:34:01Z")

</div>

I am having an issue in elasticsearch 0.90.3 where I am using logstash to  
insert data. My issue is that when I stop a node in the 3 node cluster, no  
new data is inserted and yet the status still shows as green. Which in turn  
is causing logstash to continue to send it data and elasticsearch is  
essentially dropping it on the floor.

There were 3 nodes in the cluster, ip2/es02 was the master and I shut it  
down. I then monitored the rest of the nodes, and master election occurred,  
and a new master was selected. However, no matter what I do, no data is  
flowing into the elasticsearch cluster, and there are no errors from the  
logstash side of things.

Any help on how to troubleshoot this would be great!

Some just general information:

curl [http://localhost:9200/\_nodes?pretty=true](http://localhost:9200/_nodes?pretty=true)  
{  
"ok" : true,  
"cluster\_name" : "tower3",  
"nodes" : {  
"idOfThree" : {  
"name" : "es03",  
"transport\_address" : "inet[/ip3:9300]",  
"hostname" : "es03",  
"version" : "0.90.3",  
"http\_address" : "inet[/ip3:9200]",  
"attributes" : {  
"datacenter" : "usw"  
}  
},  
"idOfOne" : {  
"name" : "es01",  
"transport\_address" : "inet[/ip1:9300]",  
"hostname" : "es01",  
"version" : "0.90.3",  
"http\_address" : "inet[/ip1:9200]",  
"attributes" : {  
"datacenter" : "usw"  
}  
}  
}  
}

curl [http://localhost:9200/\_cluster/health?pretty=true](http://localhost:9200/_cluster/health?pretty=true)  
{  
"cluster\_name" : "logstash",  
"status" : "green",  
"timed\_out" : false,  
"number\_of\_nodes" : 2,  
"number\_of\_data\_nodes" : 2,  
"active\_primary\_shards" : 5,  
"active\_shards" : 10,  
"relocating\_shards" : 0,  
"initializing\_shards" : 0,  
"unassigned\_shards" : 0  
}

My Elasticsearch config (somewhat modified, i.e. ips are real ips,  
node.name is different, cluster.name is different) This is similar minus  
the node.name being the actual hostname of the machine on all of the  
servers.

##################################################################

# /etc/elasticsearch/elasticsearch.yml

# 

# Base configuration for a write heavy cluster

# 

# Cluster / Node Basics

cluster.name: logstash

# Node can have abritrary attributes we can use for routing

node.name: es01  
node.datacenter: usw

path.data: /data/var/lib/elasticsearch/

# Force all memory to be locked, forcing the JVM to never swap

bootstrap.mlockall: true

# Indexing Settings for Writes

indices.memory.index\_buffer\_size: 50%  
index.refresh\_interval: 30  
index.translog.flush\_threshold\_ops: 50000  
index.store.compress.stored: true

## Threadpool Settings

# Search pool

threadpool.search.type: fixed  
threadpool.search.size: 20  
threadpool.search.queue\_size: 100

# Bulk pool

#threadpool.bulk.type: fixed  
#threadpool.bulk.size: 60  
#threadpool.bulk.queue\_size: 300

# Index pool

threadpool.index.type: fixed  
threadpool.index.size: 60  
threadpool.index.queue\_size: 100

# Minimum nodes alive to constitute an operational cluster (should be n/2+1

where n is the total number of nodes in a cluster)  
discovery.zen.minimum\_master\_nodes: 2

# Unicast Discovery (disable multicast)

discovery.zen.ping.multicast.enabled: false  
discovery.zen.ping.unicast.hosts: ["ip1", "ip2", "ip3"]

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Gregory\_Durham](https://avatars.discourse-cdn.com/v4/letter/g/3ec8ea/32.png) [@Gregory\_Durham](https://discuss.elastic.co/u/Gregory_Durham)\
**Post date:** [October 16, 2013, 10:39pm UTC](https://discuss.elastic.co/t/data-loss-on-shutdown-of-node-in-a-3-node-cluster/13977/2 "2013-10-16T22:39:37Z")

</div>

Hope this may come to some help to someone down the road so that they don't  
go through the same issue I did.

After doing a deep dive into my config what I found was the following:  
org.elasticsearch.discovery.zen: [Warwolves] failed to send join request to  
master ... reason [org.elasticsearch.ElasticSearchTimeoutException: Timeout  
waiting for task.]

What I found is since the client and server both lived on the same server  
there would be contention for port 9300, which meant the client, would  
start on 9301 (found this in netstat -lanp) Reading through documentation  
pointed out that this is bidirectional, which meant on the other side 9301  
had to be open and accessible. Since this is in AWS, I made a rule in the  
security-group to allow several ports above 9300 from within and this  
appears to have resolved the issue.

Thank you,  
Greg

On Wednesday, October 16, 2013 1:34:01 PM UTC-7, Gregory Durham wrote:

> I am having an issue in elasticsearch 0.90.3 where I am using logstash to  
> insert data. My issue is that when I stop a node in the 3 node cluster, no  
> new data is inserted and yet the status still shows as green. Which in turn  
> is causing logstash to continue to send it data and elasticsearch is  
> essentially dropping it on the floor.
> 
> There were 3 nodes in the cluster, ip2/es02 was the master and I shut it  
> down. I then monitored the rest of the nodes, and master election occurred,  
> and a new master was selected. However, no matter what I do, no data is  
> flowing into the elasticsearch cluster, and there are no errors from the  
> logstash side of things.
> 
> Any help on how to troubleshoot this would be great!
> 
> Some just general information:
> 
> curl [http://localhost:9200/\_nodes?pretty=true](http://localhost:9200/_nodes?pretty=true)  
> {  
> "ok" : true,  
> "cluster\_name" : "tower3",  
> "nodes" : {  
> "idOfThree" : {  
> "name" : "es03",  
> "transport\_address" : "inet[/ip3:9300]",  
> "hostname" : "es03",  
> "version" : "0.90.3",  
> "http\_address" : "inet[/ip3:9200]",  
> "attributes" : {  
> "datacenter" : "usw"  
> }  
> },  
> "idOfOne" : {  
> "name" : "es01",  
> "transport\_address" : "inet[/ip1:9300]",  
> "hostname" : "es01",  
> "version" : "0.90.3",  
> "http\_address" : "inet[/ip1:9200]",  
> "attributes" : {  
> "datacenter" : "usw"  
> }  
> }  
> }  
> }
> 
> curl [http://localhost:9200/\_cluster/health?pretty=true](http://localhost:9200/_cluster/health?pretty=true)  
> {  
> "cluster\_name" : "logstash",  
> "status" : "green",  
> "timed\_out" : false,  
> "number\_of\_nodes" : 2,  
> "number\_of\_data\_nodes" : 2,  
> "active\_primary\_shards" : 5,  
> "active\_shards" : 10,  
> "relocating\_shards" : 0,  
> "initializing\_shards" : 0,  
> "unassigned\_shards" : 0  
> }
> 
> My Elasticsearch config (somewhat modified, i.e. ips are real ips,  
> node.name is different, cluster.name is different) This is similar minus  
> the node.name being the actual hostname of the machine on all of the  
> servers.
> 
> ##################################################################
> 
> # /etc/elasticsearch/elasticsearch.yml
> 
> # 
> 
> # Base configuration for a write heavy cluster
> 
> # 
> 
> # Cluster / Node Basics
> 
> cluster.name: logstash
> 
> # Node can have abritrary attributes we can use for routing
> 
> node.name: es01  
> node.datacenter: usw
> 
> path.data: /data/var/lib/elasticsearch/
> 
> # Force all memory to be locked, forcing the JVM to never swap
> 
> bootstrap.mlockall: true
> 
> # Indexing Settings for Writes
> 
> indices.memory.index\_buffer\_size: 50%  
> index.refresh\_interval: 30  
> index.translog.flush\_threshold\_ops: 50000  
> index.store.compress.stored: true
> 
> ## Threadpool Settings
> 
> # Search pool
> 
> threadpool.search.type: fixed  
> threadpool.search.size: 20  
> threadpool.search.queue\_size: 100
> 
> # Bulk pool
> 
> #threadpool.bulk.type: fixed  
> #threadpool.bulk.size: 60  
> #threadpool.bulk.queue\_size: 300
> 
> # Index pool
> 
> threadpool.index.type: fixed  
> threadpool.index.size: 60  
> threadpool.index.queue\_size: 100
> 
> # Minimum nodes alive to constitute an operational cluster (should be
> 
> n/2+1 where n is the total number of nodes in a cluster)  
> discovery.zen.minimum\_master\_nodes: 2
> 
> # Unicast Discovery (disable multicast)
> 
> discovery.zen.ping.multicast.enabled: false  
> discovery.zen.ping.unicast.hosts: ["ip1", "ip2", "ip3"]

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 2:12am UTC](https://discuss.elastic.co/t/data-loss-on-shutdown-of-node-in-a-3-node-cluster/13977/3 "2017-07-06T02:12:04Z")

</div>