Logstash input/ouput elasticsearch plugin capped performances

[rschirin (Robert)]

Hey There,
inside a Logstash pipeline I was using both input and output elasticsearch plugin.
Taking a look to Stack monitoring I saw that Event Received Rate (/s) and Event Emitted Rate (/s) are capped to 4000 events/s.
Is there anything that I can do to go over this limit?

[elasticforme (Sachin)]

may be because you have no more events then 4000/second. I have 5000 event/second

[rschirin (Robert)]

are you using a specific value for scroll and/or size parameter in the input plugin?

[elasticforme (Sachin)]

no, nothing special.
only thing different then default is

pipeline.workers: 20
pipeline.batch.size: 256
queue.type: persisted

[rschirin (Robert)]

ok, I will try to give it a little boost with workers

let you know

[cknz (Cameron Kerr)]

If you look at the logstash API's node pipeline statistics, you may well find that the elasticsearch output is the slowest; in my case it is the slowest by a considerable margin, and would indicate that I should put more of my effort on increasing indexing speed within Elasticsearch.

There is plenty of useful information on optimizing index speed for Elasticsearch.

If you don't already have some decent visibility of the pipeline processing statistics for Elasticsearch, here's a script you might find useful (I use it for informal plugin performance testing during development).

Change the 'memcache-get' to be the 'id' of any module you use (eg. grok { id => "some_invocation_of_grok" .... } )

#!/bin/bash

poll_interval=1 # seconds

get_stats() {
    while true
    do
        curl -s 127.0.0.1:9600/_node/stats/pipelines/main \
            | jq -r '.pipelines.main.plugins.filters[] | select(.id == "memcached-get") | "\(.events.duration_in_millis) \(.events.out)"'
        sleep $poll_interval
    done
}

get_stats | awk -v poll_interval=$poll_interval '
    NR == 1 {
        last_total_duration_millis = $1;
        last_total_events = $2;
        next;
    }

    NR > 1 {
        duration_millis_delta = $1 - last_total_duration_millis;
        events_delta = $2 - last_total_events;
        # For each event, how much time was spent, in microseconds

        if (events_delta == 0) {
            usps = "-"
        } else {
            usps = duration_millis_delta*1000 / poll_interval / events_delta;
        }
        keps = events_delta / poll_interval / 1000.0;

        printf("%.2f μs per event, %.1f keps\n", usps, keps);

        last_total_duration_millis = $1;
        last_total_events = $2;
    }
'