# Elasticsearch stops ingesting data when node is down

**URL:** <https://discuss.elastic.co/t/elasticsearch-stops-ingesting-data-when-node-is-down/131804>\
**Category:** Elasticsearch\
**Created:** [May 14, 2018, 6:51pm UTC](https://discuss.elastic.co/t/elasticsearch-stops-ingesting-data-when-node-is-down/131804 "2018-05-14T18:51:48Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![igor.leao1](https://avatars.discourse-cdn.com/v4/letter/i/9e8a1a/32.png) [@igor.leao1](https://discuss.elastic.co/u/igor.leao1)\
**Post date:** [May 14, 2018, 6:51pm UTC](https://discuss.elastic.co/t/elasticsearch-stops-ingesting-data-when-node-is-down/131804/1 "2018-05-14T18:51:48Z")

</div>

Hi there,

I have an ES cluster which stops ingesting data every time a node is down.  
As it is a cluster used for debugging purposes only, it retains logs has no replication for most of its shards. There are 13 data nodes (i3.3xlarge) and 3 dedicated masters (m4.large).

Also, it seems that ingestion rate decreases everytime shards get reallocated, as well as when a certain node faces high cpu usage.

Any idea what may be happening?

Thanks!

**Summary**  
[root@ip-10-0-0-212 elasticsearch]# curl localhost:9200  
{  
"name" : "tfg-es-logs-cluster-node-x",  
"cluster\_name" : "my-cluster",  
"cluster\_uuid" : my-uid",  
"version" : {  
"number" : "5.6.5",  
"build\_hash" : "6a37571",  
"build\_date" : "2017-12-04T07:50:10.466Z",  
"build\_snapshot" : false,  
"lucene\_version" : "6.6.1"  
},  
"tagline" : "You Know, for Search"  
}

**Cluster health:**  
[ec2-user@ip-10-x-x-x ~]$ curl localhost:9200/\_cluster/health?pretty  
{  
"cluster\_name" : "my-cluster",  
"status" : "green",  
"timed\_out" : false,  
"number\_of\_nodes" : 16,  
"number\_of\_data\_nodes" : 13,  
"active\_primary\_shards" : 8135,  
"active\_shards" : 8358,  
"relocating\_shards" : 1,  
"initializing\_shards" : 0,  
"unassigned\_shards" : 0,  
"delayed\_unassigned\_shards" : 0,  
"number\_of\_pending\_tasks" : 0,  
"number\_of\_in\_flight\_fetch" : 0,  
"task\_max\_waiting\_in\_queue\_millis" : 0,  
"active\_shards\_percent\_as\_number" : 100.0  
}

**SO:**  
Amazon Linux  
[ec2-user@ip-10-0-0-212 ~]$ uname -a  
Linux ip-10-0-0-212 4.9.77-31.58.amzn1.x86\_64 #1 SMP Thu Jan 18 22:15:23 UTC 2018 x86\_64 x86\_64 x86\_64 GNU/Linux

**Elasticsearch.yml**  
cluster.name: my-clyster  
node.name: my-cluster-my-node  
path.conf: "/etc/elasticsearch"  
path.data: "/mnt/es\_disk"  
path.logs: "/var/log/elasticsearch"  
network.host: 0.0.0.0  
discovery.type: ec2  
discovery.ec2.groups: sg-my-sg  
discovery.ec2.any\_group: false  
cloud.node.auto\_attributes: true  
bootstrap.memory\_lock: true  
script.inline: true  
script.stored: true  
xpack.security.enabled: false  
xpack.monitoring.enabled: false  
node.data: true  
node.master: false  
node.ingest: true  
node.attr.index\_type: log

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [May 14, 2018, 9:21pm UTC](https://discuss.elastic.co/t/elasticsearch-stops-ingesting-data-when-node-is-down/131804/2 "2018-05-14T21:21:41Z")

</div>

If you do not have a replica configured for an index, the index will go red if you lose a shard, which means you can not index into it. If you did have a replica, Elasticsearch would promote the replica to primary if you lose one, and you can continue indexing into it as there is at least one copy of each shard.

---

<div class="post-metadata">

**Author:** ![igor.leao1](https://avatars.discourse-cdn.com/v4/letter/i/9e8a1a/32.png) [@igor.leao1](https://discuss.elastic.co/u/igor.leao1)\
**Post date:** [May 14, 2018, 10:45pm UTC](https://discuss.elastic.co/t/elasticsearch-stops-ingesting-data-when-node-is-down/131804/3 "2018-05-14T22:45:40Z")

</div>

Thanks @Christian_Dahlqvist .

Let's imagine node X is down for a few minutes. Node X contains primary shards px1, px2 and px3. We also have node Y with primary shards py1, py2 and py3. If Node X is down, should this affect ingestion of py1, py2 and py3? (considering both scenarios where px\_i and py\_i belong to the same and different indices)

---

<div class="post-metadata">

**Author:** ![Johnnycc1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johnnycc1/32/26069_2.png) [@Johnnycc1](https://discuss.elastic.co/u/Johnnycc1)\
**Post date:** [May 14, 2018, 11:36pm UTC](https://discuss.elastic.co/t/elasticsearch-stops-ingesting-data-when-node-is-down/131804/4 "2018-05-14T23:36:50Z")

</div>

If they belong to the same index then indexing should stop - as there will be documents being indexed that need to be allocated to primary shards that are not available.

If they belong to different indexes then happy days.

I recommend turning on compression on the indexes and replicating the data. Data lost is not fun even if the data is just logs.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 11, 2018, 11:36pm UTC](https://discuss.elastic.co/t/elasticsearch-stops-ingesting-data-when-node-is-down/131804/5 "2018-06-11T23:36:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.