# Retrying failed action with response code: 429

**URL:** <https://discuss.elastic.co/t/retrying-failed-action-with-response-code-429/34679>\
**Category:** Logstash\
**Created:** [November 16, 2015, 1:29pm UTC](https://discuss.elastic.co/t/retrying-failed-action-with-response-code-429/34679 "2015-11-16T13:29:44Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![jsosic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsosic/32/5945_2.png) [@jsosic](https://discuss.elastic.co/u/jsosic)\
**Post date:** [November 16, 2015, 1:29pm UTC](https://discuss.elastic.co/t/retrying-failed-action-with-response-code-429/34679/1 "2015-11-16T13:29:44Z")

</div>

Hi guys,

I get a lot of the messages in logstash log that look like:

`
retrying failed action with response code: 429
`

I'm using redis as input, and elasticsearch as output and my logstash.conf is pretty straightforward:

`
input {
    redis {
        host => "127.0.0.1"
        data_type => "list"
        key => "logstash"
        codec => json
        threads => 16
    }
}
output {
elasticsearch {
hosts => "127.0.0.1:9200"
index => "logstash-%{app}-%{+YYYY.MM.dd}"
codec => "plain"
workers => 16
flush_size => 1000
idle_flush_time => 1
}
}
`

Any ideas what could be the cause for these errors?

ElasticSearch has 16GB of RAM available out of 32 on the system, IO is pretty small, cpu load on VM is also pretty low (\<0.5)...

Also, I'm not seeing all the logs I'm expecting to see in the kibana, and don't know if this could be the cause for that, or maybe I have issues on the agent side.

I'm running these versions:

ElasticSearch 1.5.2  
LogStash: 2.0.0 (was running 1.5.x with same problems)

How can I debug this further?

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [November 16, 2015, 2:51pm UTC](https://discuss.elastic.co/t/retrying-failed-action-with-response-code-429/34679/2 "2015-11-16T14:51:51Z")

</div>

> [@jsosic](#):
>
> codec =\> "plain"  
> workers =\> 16  
> flush\_size =\> 1000  
> idle\_flush\_time =\> 1

I would remove these 4 lines and see what happens with the defaults.

16 workers is generally far more than necessary. I wouldn't make this setting more than 2 unless you're doing over 10,000 events per second. Flush size is also too big. Because of the retry logic (which is why you get 429 response codes), you should probably work in smaller batches (I believe the default is 512 now). The plain codec simply doesn't do anything here, as elasticsearch _requires_ JSON.

I also note that you are separating your indices by app. How many "apps" do you have per day? How many indices do you have, total, on your cluster? What's your data retention policy? Are you using the default 5+1 shard count? I ask these questions because having too many shards on a single node can overload a node's index management ability. It only gets to use a percentage of the heap for this, and exhausting the memory creates pressure which can dramatically affect index caching (which might be what's resulting in more 429s).

---

<div class="post-metadata">

**Author:** ![jsosic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsosic/32/5945_2.png) [@jsosic](https://discuss.elastic.co/u/jsosic)\
**Post date:** [November 16, 2015, 4:27pm UTC](https://discuss.elastic.co/t/retrying-failed-action-with-response-code-429/34679/3 "2015-11-16T16:27:00Z")

</div>

I have around 720 shards. I keep number\_of\_shards: 5 and number\_of\_replicas at 0. I have only 1 node in my ES cluster.

I've removed those lines and so far I don't see anything in logstash logs... I dropped all of my indexes and am now reindexing logs...

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [November 16, 2015, 4:29pm UTC](https://discuss.elastic.co/t/retrying-failed-action-with-response-code-429/34679/4 "2015-11-16T16:29:23Z")

</div>

720 shards is quite a few for a single node, as is 5 shards per index on a single node. For a single node, I would suggest that you only need 1 shard, maybe 2.

How many "apps" do you have per day (i.e., how many indices are created each day)? This number will have a profound impact as it dictates the number of "active" shards, which want _more_ of the index cache.

---

<div class="post-metadata">

**Author:** ![jsosic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsosic/32/5945_2.png) [@jsosic](https://discuss.elastic.co/u/jsosic)\
**Post date:** [November 16, 2015, 10:18pm UTC](https://discuss.elastic.co/t/retrying-failed-action-with-response-code-429/34679/5 "2015-11-16T22:18:16Z")

</div>

OK, I lowered it to 1 shard per indice and I'm down to 51 shards. I keep logs for 10 days, create indices daily and have approximately 5-6 apps.

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [November 16, 2015, 10:31pm UTC](https://discuss.elastic.co/t/retrying-failed-action-with-response-code-429/34679/6 "2015-11-16T22:31:08Z")

</div>

Those numbers are much more tenable. You shouldn't exhaust your index cache with shard counts like those.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:22am UTC](https://discuss.elastic.co/t/retrying-failed-action-with-response-code-429/34679/7 "2017-07-06T05:22:39Z")

</div>