Hi,
We would like to use Logstash to receive log messages from multiple services (nxlog) and send them further to Elastic.
i.e. using this architecture - but with nxlog instead of Beats:
What are the limits of a single Logstash instance if used as described above?
It is impossible to say. The throughput you can achieve in a single logstash instance may vary by multiple orders of magnitude (i.e. 1000x or more) depending on the hardware configuration and the amount and type of processing you are doing.
The best option is to test. Avoid grok or any other slower plugin or algorithm.
Echoing what @rios said, test, I would recommend testing on a small/medium/large node because you won’t know what the max throughput your logstash configuration can handle is without thorough testing. Heck you can’t even speculate what the throughput might be without testing.
I’ve seen things cap out at 7K eps, but then I’ve also seen 100K eps, but there are many, many variables to consider, so I would say keep your test cases simple and see what kind of results you get. A very very oversimplified example
Etc etc… the challenging part will be generating load from nxlog depending on your input, if it’s something simple like tcp/udp, you may not even need to actually use nxlog if you know the structure of your logs. For example when testing something like udp I’ve had a lot of success just using 1+ logstash nodes with a generator input shipping tcp/udp logs (stressor) to a second logstash (the system being tested)
And you can use LS statistic and Kibana LS monitoring in order to get info per plugin.
curl -XGET 'localhost:9600/_node/stats/pipelines?pretty'
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.
