Logstash Limits

Elastic Stack Logstash
1

Hi,
We would like to use Logstash to receive log messages from multiple services (nxlog) and send them further to Elastic.
i.e. using this architecture - but with nxlog instead of Beats:

image
image1454×940 89 KB

What are the limits of a single Logstash instance if used as described above?

2

It is impossible to say. The throughput you can achieve in a single logstash instance may vary by multiple orders of magnitude (i.e. 1000x or more) depending on the hardware configuration and the amount and type of processing you are doing.

3

The best option is to test. Avoid grok or any other slower plugin or algorithm.

4

Echoing what @rios said, test, I would recommend testing on a small/medium/large node because you won’t know what the max throughput your logstash configuration can handle is without thorough testing. Heck you can’t even speculate what the throughput might be without testing.

I’ve seen things cap out at 7K eps, but then I’ve also seen 100K eps, but there are many, many variables to consider, so I would say keep your test cases simple and see what kind of results you get. A very very oversimplified example

small node 2G - JVM - default settings for batch size and workers etc

  1. 1 x upstream input source
  2. 2 x upstream input source
  3. 4 x upstream input source

medium node 4G - JVM - default settings for batch size and workers etc

  1. 1 x upstream input source
  2. 2 x upstream input source
  3. 4 x upstream input source

Etc etc… the challenging part will be generating load from nxlog depending on your input, if it’s something simple like tcp/udp, you may not even need to actually use nxlog if you know the structure of your logs. For example when testing something like udp I’ve had a lot of success just using 1+ logstash nodes with a generator input shipping tcp/udp logs (stressor) to a second logstash (the system being tested)

1 Like
5

And you can use LS statistic and Kibana LS monitoring in order to get info per plugin.

curl -XGET 'localhost:9600/_node/stats/pipelines?pretty'

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.