Logstash input/ouput limits per/sec

Elastic Stack Logstash
1

hi everyone.

could someone point out the logstash limit, plz?

let me say my scenario first.

i use cloudwatch_logs input > do 50-line grok match filter > elasticsearch output

base on my script, i would like to know few facts:

  1. cloudwatch_logs input streaming limit per/sec ?
  2. elasticsearch output streaming limit per/sec ?
  3. if many log events come in same sec, this may cause performance issue in logstash and kibana searching? may cause lost logs?
  4. using 50-line grok match filter cause slow the logstash performance?

I'm running logstash v5.6.3.

2

cloudwatch_logs input streaming limit per/sec ?

That depends on several factors, including the performance of the host, filters used, etc.

elasticsearch output streaming limit per/sec ?

See above.

if many log events come in same sec, this may cause performance issue in logstash and kibana searching? may cause lost logs?

If you enable the persistent queue you shouldn't lose any events. Even without the persistent queue you'll be fine if you only have inputs that handle backpressure well. The file input, for example, will just stop reading from the input file and in that sense has its own queue system while the udp input needs to deal with whatever gets sent to it.

using 50-line grok match filter cause slow the logstash performance?

Large grok filters will of course be detrimental to performance but whether that actually matters for you is another story.

2 Likes
3

Thank you magnusbaeck.

In case of using large grok filter, what should i do not to be detrimental to performance?

4

This blog post provides some very good guidelines on how to best use grok. In addition to this I would add that it helps being aware of other types of filters, so you do not try to use grok to parse content where better and more efficient options exist, e.g. lists of key-value pairs and JSON content.

1 Like
5

Thanks for your help.
Your post is good to help me to take care of regex pattern and regex engine behavior.
kv filter is useful for me as well. :slight_smile:

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.