Many http error 429s in Logstash

Elastic Stack Logstash
1

I'm a bit new to ELK but have a environment that sends high-volume and bursty logs in the following configuration:

Fluentd (around 15 instances) -> (via HTTP) -> LogStash (1 containerized instance) -> AWS ElasticSearch (3 node)

I'm currently seeing my Fluentd instances fail to send logs to LogStash with multiple http error 429s. Once this happens, I no longer see any results show up in Kibana. Searching online seems to tell me that this is a problem with indexing at the ElasticSearch stage not keeping up. I would like to understand what my options are to resolve this.

2

How many shards are you actively indexing into? How many workers does your Logstash instance use? This blog post might help explain what is going on.

3

Hi Christian, thanks for your response. I think I may have misdiagnosed my issue. It seems that when Fluentd tries to flush its chunks to Logstash, it runs into a timeout. I don't see any errors on my Logstash instance and am running with DEBUG so not sure how I can figure out the cause of the timeout.

Additionally, I see that the console log for Logstash shows the ruby debug output for events that were emitted over 30 minutes ago. So it seems like something is backed up on the Logstash side.

Any suggestions on where to look or how to debug the issue?

4

What is the specification of your Elasticsearch cluster? What type of instances are you using?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.