Speed up when logstash is running

Elastic Stack Logstash
1

Question1:
My Current CPU utilization is only 10% in my 36-core aws server, how can i increase it to say 80%?
I am using logstash and ES 5.1.1. ES is a standard server with 4 nodes.
Input is S3 buckets, with thousands of files of 2-3G. Output is the ES server.
Shall I increase worker in logstash.yml file?

Question2:
When I do -b 800 -w 70, it always gives me this warninng. How should I deal with it?
CAUTION: Recommended inflight events max exceeded! Logstash will run with up to 18000 events in memory in your current configuration. If your message sizes are large this may cause instability with the default heap size. Please consider setting a non-standard heap size, changing the batch size (currently 500), or changing the number of pipeline workers (currently 36)

2

How do you know Logstash is the bottleneck? What kind of inputs and outputs do you have? Logstash is most likely just one piece of the chain and the slowest component is going to dictate the throughput for the other ones.

1 Like
3

Hi I add the input/output in the original post.

I am not saying it is a bottleneck, I want to know what way are there to speed it up even more, given that it is currently only 10% CPU utilization?
Whenever I increase the worker (to be>36, since I have 36 core machine) or batch size, it gives me the caution. Shall I ignore it?

4

My point is that you have to view your event pipeline as one system. If you've maxed out ES no amount of Logstash tweaking is going to help. You need to locate where your current bottleneck is and focus your attention there. When you're done the bottleneck might be elsewhere, so you shift your focus. Repeat until you're happy.

1 Like
5

This is what I get from _node/stats/pipeline, Could I tell if ES output is maxes out?
How can i increase the size 5239k?
{ "host":"ip-10-******", "version":"5.1.1", "http_address":"127.0.0.1:9600", "id":"7974547c-fc36-4eae-8e22-a5ffc3496923", "name":"ip-10-*********", "pipeline":{ "events":{ "duration_in_millis":140652688, "in":5239913, "filtered":5239663, "out":5236564 }, "plugins":{ "inputs":[ ], "filters":[ { "id":"7a672e3d664774e7f6ad76cfab2cb1ed895f163c-2", "events":{ "duration_in_millis":519023, "in":5239776, "out":5239775 }, "matches":5239776, "patterns_per_field":{ "message":1 }, "name":"grok" }, { "id":"7a672e3d664774e7f6ad76cfab2cb1ed895f163c-4", "events":{ "duration_in_millis":101482, "in":5239775, "out":5239775 }, "name":"date" }, { "id":"7a672e3d664774e7f6ad76cfab2cb1ed895f163c-7", "events":{ "duration_in_millis":811219, "in":5239775, "out":5239775 }, "name":"kv" }, { "id":"7a672e3d664774e7f6ad76cfab2cb1ed895f163c-9", "events":{ "duration_in_millis":179229, "in":5239775, "out":5239774 }, "name":"mutate" }, { "id":"7a672e3d664774e7f6ad76cfab2cb1ed895f163c-11", "events":{ "duration_in_millis":724520, "in":5239774, "out":5239774 }, "name":"geoip" }, { "id":"7a672e3d664774e7f6ad76cfab2cb1ed895f163c-8", "events":{ "duration_in_millis":2551609, "in":5239775, "out":5239775 }, "matches":643597, "failures":4596178, "patterns_per_field":{ "tag_content":1, "url":2 }, "name":"grok" }, { "id":"7a672e3d664774e7f6ad76cfab2cb1ed895f163c-12", "events":{ "duration_in_millis":594548, "in":5239774, "out":5239774 }, "name":"prune" }, { "id":"7a672e3d664774e7f6ad76cfab2cb1ed895f163c-6", "events":{ "duration_in_millis":150492, "in":5239775, "out":5239775 }, "name":"mutate" }, { "id":"7a672e3d664774e7f6ad76cfab2cb1ed895f163c-3", "events":{ "duration_in_millis":282718, "in":5239776, "out":5239776 }, "name":"json" }, { "id":"7a672e3d664774e7f6ad76cfab2cb1ed895f163c-5", "events":{ "duration_in_millis":169727, "in":5239776, "out":5239775 }, "name":"urldecode" }, { "id":"7a672e3d664774e7f6ad76cfab2cb1ed895f163c-10", "events":{ "duration_in_millis":142807, "in":5239775, "out":5239775 }, "name":"mutate" } ], "outputs":[ { "id":"7a672e3d664774e7f6ad76cfab2cb1ed895f163c-13", "events":{ "in":5239663, "out":5239663 }, "name":"amazon_es" } ] }

6

Could I tell if ES output is maxes out?

No. To figure out the capacity of ES you should measure ES alone. Use an existing benchmarking tool (maybe Rally?) or roll your own small script and measure what rate ES is capable of receiving.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.