Hey There,
inside a Logstash pipeline I was using both input and output elasticsearch
plugin.
Taking a look to Stack monitoring I saw that Event Received Rate (/s) and Event Emitted Rate (/s) are capped to 4000 events/s.
Is there anything that I can do to go over this limit?
are you using a specific value for scroll
and/or size
parameter in the input plugin?
no, nothing special.
only thing different then default is
pipeline.workers: 20
pipeline.batch.size: 256
queue.type: persisted
ok, I will try to give it a little boost with workers
let you know
If you look at the logstash API's node pipeline statistics, you may well find that the elasticsearch output is the slowest; in my case it is the slowest by a considerable margin, and would indicate that I should put more of my effort on increasing indexing speed within Elasticsearch.
There is plenty of useful information on optimizing index speed for Elasticsearch.
If you don't already have some decent visibility of the pipeline processing statistics for Elasticsearch, here's a script you might find useful (I use it for informal plugin performance testing during development).
Change the 'memcache-get' to be the 'id' of any module you use (eg. grok { id => "some_invocation_of_grok" .... } )
#!/bin/bash
poll_interval=1 # seconds
get_stats() {
while true
do
curl -s 127.0.0.1:9600/_node/stats/pipelines/main \
| jq -r '.pipelines.main.plugins.filters[] | select(.id == "memcached-get") | "\(.events.duration_in_millis) \(.events.out)"'
sleep $poll_interval
done
}
get_stats | awk -v poll_interval=$poll_interval '
NR == 1 {
last_total_duration_millis = $1;
last_total_events = $2;
next;
}
NR > 1 {
duration_millis_delta = $1 - last_total_duration_millis;
events_delta = $2 - last_total_events;
# For each event, how much time was spent, in microseconds
if (events_delta == 0) {
usps = "-"
} else {
usps = duration_millis_delta*1000 / poll_interval / events_delta;
}
keps = events_delta / poll_interval / 1000.0;
printf("%.2f μs per event, %.1f keps\n", usps, keps);
last_total_duration_millis = $1;
last_total_events = $2;
}
'
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.