# LogStash throwing error 503

**URL:** https://discuss.elastic.co/t/logstash-throwing-error-503/142163
**Category:** Logstash
**Created:** [July 30, 2018, 11:40am UTC](https://discuss.elastic.co/t/logstash-throwing-error-503/142163 "2018-07-30T11:40:00Z")
**Posts on this page:** 10
**Page:** 1

<div class="post-metadata">

### Author: ![rmanju2020](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rmanju2020/32/44560_2.png) [@rmanju2020](https://discuss.elastic.co/u/rmanju2020)
#### Post date: [July 30, 2018, 11:40am UTC](https://discuss.elastic.co/t/logstash-throwing-error-503/142163/1 "2018-07-30T11:40:00Z")

</div>

The Logastash loggers are throwing below errors continuously.

"retrying failed action with response code: 503", :level=\>:warn}  
{:timestamp=\>"2018-07-30T16:58:20.262000+0530", :message=\>"too many attempts at sending event. dropping:  
The Elastic-search loggers doesn't have the any errors.

Logstash - 1.7.0 version  
Elastic search 1.6.0 version

The health is RED status  
Kindly guide to resolve

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 30, 2018, 12:08pm UTC](https://discuss.elastic.co/t/logstash-throwing-error-503/142163/2 "2018-07-30T12:08:24Z")

</div>

Why is the cluster in RED state? What does the [cluster health API](https://www.elastic.co/guide/en/elasticsearch/reference/1.6/cluster-health.html)?

---

<div class="post-metadata">

### Author: ![rmanju2020](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rmanju2020/32/44560_2.png) [@rmanju2020](https://discuss.elastic.co/u/rmanju2020)
#### Post date: [July 30, 2018, 12:20pm UTC](https://discuss.elastic.co/t/logstash-throwing-error-503/142163/3 "2018-07-30T12:20:19Z")

</div>

Cluster API health is as below

status "red"  
timed\_out false  
number\_of\_nodes 5  
number\_of\_data\_nodes 1  
active\_primary\_shards 704  
active\_shards 704  
relocating\_shards 0  
initializing\_shards 0  
unassigned\_shards 955  
number\_of\_pending\_tasks 0  
number\_of\_in\_flight\_fetch 0

As per my knowledge few indices are in RED state and internally those indices few shards are in UNASSIGNED state.Further i have not much knowledge to debug.

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 30, 2018, 12:27pm UTC](https://discuss.elastic.co/t/logstash-throwing-error-503/142163/4 "2018-07-30T12:27:10Z")

</div>

How much heap do you have on your data node? It would seem you have far too many shards for a cluster that size. Please read [this blog post for guidance on shards and sharding](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster).

---

<div class="post-metadata">

### Author: ![rmanju2020](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rmanju2020/32/44560_2.png) [@rmanju2020](https://discuss.elastic.co/u/rmanju2020)
#### Post date: [July 30, 2018, 12:37pm UTC](https://discuss.elastic.co/t/logstash-throwing-error-503/142163/5 "2018-07-30T12:37:11Z")

</div>

HW:16 GB RAM , 8 core windows 2008 R2 OS  
Elastic search : 6GB allocated  
LogStash: 2GB allocated  
Harddisk:1TB total, 550 GB free

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 30, 2018, 12:41pm UTC](https://discuss.elastic.co/t/logstash-throwing-error-503/142163/6 "2018-07-30T12:41:39Z")

</div>

You have far too many shards for that size of heap and need to reduce that significantly or scale the cluster up/out in order to get a stable cluster. As you are on such an old version this will require you to delete and/or reindex data if you can not increase resources.

---

<div class="post-metadata">

### Author: ![rmanju2020](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rmanju2020/32/44560_2.png) [@rmanju2020](https://discuss.elastic.co/u/rmanju2020)
#### Post date: [July 30, 2018, 12:48pm UTC](https://discuss.elastic.co/t/logstash-throwing-error-503/142163/7 "2018-07-30T12:48:18Z")

</div>

Thanks . Can you suggest on how much resource need to meet the 120% of current data.I will do it..

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 30, 2018, 12:51pm UTC](https://discuss.elastic.co/t/logstash-throwing-error-503/142163/8 "2018-07-30T12:51:58Z")

</div>

Even if you add resources you probably need to change your sharding policy and reduce the shard count substantially. having so many small shards is very inefficient. I do not have any recommendation about how much resources to add as I have not used this version in years, But adding another data node of the same size while setting the number of replicas to 0 might give you enough headroom to restructure your shards.

---

<div class="post-metadata">

### Author: ![rmanju2020](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rmanju2020/32/44560_2.png) [@rmanju2020](https://discuss.elastic.co/u/rmanju2020)
#### Post date: [July 30, 2018, 12:57pm UTC](https://discuss.elastic.co/t/logstash-throwing-error-503/142163/9 "2018-07-30T12:57:15Z")

</div>

Thank you sir,  
These actions are going to perform and let you know.  
Reduce the shards/index from 3 to 1  
Delete the the index of 3 months older  
Replica size to zero

Still Instead of adding one more node, I hope some increase resource might help.Suggest on this irrespective of version.Its not a problem and a kind of learning for us.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 27, 2018, 12:57pm UTC](https://discuss.elastic.co/t/logstash-throwing-error-503/142163/10 "2018-08-27T12:57:18Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
