Congestion issues in Redis coming from Logstash

Elastic Stack Logstash
1

Hello, I am running LS 5.3 with 5 nodes load balanced and ES 5.3, using Redis to throttle events. I am receiving the error message below within Redis, I believe this is coming from Logstash but I'm not sure how to resolve.

[2019-02-11T11:49:29,923][WARN ][logstash.outputs.redis ] Redis key size has hit a congestion threshold 2200000 suspending output for 1 seconds

Here is my input LS config file in redis:

input {
  beats {
    port => "5043"
    tags => ["aemlogs","fglam"]
    type => "linux-logs"
  }
}
output {
  if ("aemlogs" in [tags]) {
    redis {
      data_type => "list"
      key => "linux-beats"
      congestion_threshold => "2200000"
    }
  }

  if ("fglam" in [tags]) {
    redis {
      data_type => "list"
      key => "linux-beats"
      congestion_threshold => "2200000"
    }
  }
}

This is my logstash collector config file:

input {
  redis {
    type => "linux-logs"
    data_type => "list"
    key => "linux-beats"
    port => "6379"
    host => "redis-host"
  }
}
filter {
    if ("aemlogs" in [tags]) {
        grok {
          timeout_millis => 5000
          tag_on_failure => [ "_grokparsefailure","_fglamparsefailure" ]
          match => { "message" => "^%{TIMESTAMP_ISO8601:timestamp}%{SPACE}((%{SEV:log-level}%{SPACE}\[%{DATASET1:dataset1}\])?)((%{WORD}%{SPACE}\<(%{UUID:uuid}\/)?%{WORD:namespace}\/%{VERSION:version}\/%{WORD:agent_type}\/%{AGENT:agent}\>)?)%{SPACE}%{SEV:log-level}%{SPACE}\[%{DATASET1:dataset1}\]%{SPACE}%{DATASET2:dataset2}(\s-\s)%{GREEDYDATA:message}" }
          overwrite => [ "message" ]
        }
        date {
            match => [ "timestamp", "YYYY-MM-dd HH:mm:ss.SSS","ISO8601" ]
            target => "@timestamp"
            remove_field => [ "timestamp" ]
        }
    }
}
2

The message is coming from the Redis Output Plugin and indicates the the key in redis that it is configured to send data to has more than the configured number of items in it (e.g., whatever is supposed to consume from this list is slower than the things that are producing to that key).

From the docs:

In case Redis data_type is list and has more than congestion_threshold items, block until someone consumes them and reduces congestion, otherwise if there are no consumers Redis will run out of memory, unless it was configured with OOM protection. But even with OOM protection, a single Redis list can block all other users of Redis, until Redis CPU consumption reaches the max allowed RAM size. A default value of 0 means that this limit is disabled. Only supported for list Redis data_type .

-- Logstash: Redis Output Plugin - congestion_threshold

3

Thanks for the reply Ry, so does this mean that I can temporarily set the threshold to; congestion_treshold => “0” in order to avoid throttling the events?

Also, where did you find the docs for Redis errors?

Dion Rivera

Esri Information Systems and Technology |
drivera@esri.com | 909-793-2853 x1860

4

The congestion threshold is a safeguard to prevent you from losing data, as is likely to occur when Redis runs out of memory. While it is possible to remove the safeguard, the correct solution is to ensure that whatever is supposed to be consuming from that list has the resources it needs to consume as fast as you are producing.

5

I am not sure what you are asking here. I linked to the relevant documentation, which I found by plugging "Logstash Redis Output" into my search engine.

6

Thanks for the help Ry.

Dion Rivera

Esri Information Systems and Technology |
drivera@esri.com | 909-793-2853 x1860

7

Update on this issue: As @yaauie stated above, there was nothing on the LS side consuming events coming from the Redis key which caused them to pile up and throw the "Congestion" error. The LS config pipeline that was supposed to consume events coming from the Redis key was accidentally deleted. Once recovered, problem went away.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.