# Logstash input/ouput elasticsearch plugin capped performances

**URL:** <https://discuss.elastic.co/t/logstash-input-ouput-elasticsearch-plugin-capped-performances/273686>\
**Category:** Logstash\
**Created:** [May 21, 2021, 3:44pm UTC](https://discuss.elastic.co/t/logstash-input-ouput-elasticsearch-plugin-capped-performances/273686 "2021-05-21T15:44:45Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![rschirin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rschirin/32/45283_2.png) [@rschirin](https://discuss.elastic.co/u/rschirin)\
**Post date:** [May 21, 2021, 3:44pm UTC](https://discuss.elastic.co/t/logstash-input-ouput-elasticsearch-plugin-capped-performances/273686/1 "2021-05-21T15:44:45Z")

</div>

Hey There,  
inside a Logstash pipeline I was using both input and output `elasticsearch` plugin.  
Taking a look to Stack monitoring I saw that Event Received Rate (/s) and Event Emitted Rate (/s) are capped to 4000 events/s.  
Is there anything that I can do to go over this limit?

 ![Screenshot 2021-05-21 at 17.47.39](https://us1.discourse-cdn.com/elastic/original/3X/b/4/b40ae102a14f275a811533a183c5f03f21f8d43d.png)

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [May 21, 2021, 5:06pm UTC](https://discuss.elastic.co/t/logstash-input-ouput-elasticsearch-plugin-capped-performances/273686/2 "2021-05-21T17:06:35Z")

</div>

may be because you have no more events then 4000/second. I have 5000 event/second

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/7/276e9c85949cba02e089e4c2fb51229834a5f9fb.png)

---

<div class="post-metadata">

**Author:** ![rschirin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rschirin/32/45283_2.png) [@rschirin](https://discuss.elastic.co/u/rschirin)\
**Post date:** [May 21, 2021, 6:22pm UTC](https://discuss.elastic.co/t/logstash-input-ouput-elasticsearch-plugin-capped-performances/273686/3 "2021-05-21T18:22:18Z")

</div>

are you using a specific value for `scroll` and/or `size` parameter in the input plugin?

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [May 21, 2021, 6:25pm UTC](https://discuss.elastic.co/t/logstash-input-ouput-elasticsearch-plugin-capped-performances/273686/4 "2021-05-21T18:25:10Z")

</div>

no, nothing special.  
only thing different then default is

pipeline.workers: 20  
pipeline.batch.size: 256  
queue.type: persisted

---

<div class="post-metadata">

**Author:** ![rschirin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rschirin/32/45283_2.png) [@rschirin](https://discuss.elastic.co/u/rschirin)\
**Post date:** [May 21, 2021, 6:26pm UTC](https://discuss.elastic.co/t/logstash-input-ouput-elasticsearch-plugin-capped-performances/273686/5 "2021-05-21T18:26:12Z")

</div>

ok, I will try to give it a little boost with `workers`

let you know

---

<div class="post-metadata">

**Author:** ![cknz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cknz/32/9640_2.png) [@cknz](https://discuss.elastic.co/u/cknz)\
**Post date:** [May 22, 2021, 11:55am UTC](https://discuss.elastic.co/t/logstash-input-ouput-elasticsearch-plugin-capped-performances/273686/6 "2021-05-22T11:55:22Z")

</div>

If you look at the logstash API's node pipeline statistics, you may well find that the elasticsearch output is the slowest; in my case it is the slowest by a considerable margin, and would indicate that I should put more of my effort on increasing indexing speed within Elasticsearch.

There is plenty of useful information on optimizing index speed for Elasticsearch.

If you don't already have some decent visibility of the pipeline processing statistics for Elasticsearch, here's a script you might find useful (I use it for informal plugin performance testing during development).

Change the 'memcache-get' to be the 'id' of any module you use (eg. grok { id =\> "some\_invocation\_of\_grok" .... } )

```auto
#!/bin/bash

poll_interval=1 # seconds

get_stats() {
    while true
    do
        curl -s 127.0.0.1:9600/_node/stats/pipelines/main \
            | jq -r '.pipelines.main.plugins.filters[] | select(.id == "memcached-get") | "\(.events.duration_in_millis) \(.events.out)"'
        sleep $poll_interval
    done
}

get_stats | awk -v poll_interval=$poll_interval '
    NR == 1 {
        last_total_duration_millis = $1;
        last_total_events = $2;
        next;
    }

    NR > 1 {
        duration_millis_delta = $1 - last_total_duration_millis;
        events_delta = $2 - last_total_events;
        # For each event, how much time was spent, in microseconds

        if (events_delta == 0) {
            usps = "-"
        } else {
            usps = duration_millis_delta*1000 / poll_interval / events_delta;
        }
        keps = events_delta / poll_interval / 1000.0;

        printf("%.2f μs per event, %.1f keps\n", usps, keps);

        last_total_duration_millis = $1;
        last_total_events = $2;
    }
'

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 19, 2021, 11:56am UTC](https://discuss.elastic.co/t/logstash-input-ouput-elasticsearch-plugin-capped-performances/273686/7 "2021-06-19T11:56:20Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.