# Logstash-Filter-Elasticsearch Slow

**URL:** <https://discuss.elastic.co/t/logstash-filter-elasticsearch-slow/110583>\
**Category:** Logstash\
**Created:** [December 6, 2017, 11:38pm UTC](https://discuss.elastic.co/t/logstash-filter-elasticsearch-slow/110583 "2017-12-06T23:38:32Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Wayne\_Taylor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wayne_taylor/32/45984_2.png) [@Wayne\_Taylor](https://discuss.elastic.co/u/Wayne_Taylor)\
**Post date:** [December 6, 2017, 11:38pm UTC](https://discuss.elastic.co/t/logstash-filter-elasticsearch-slow/110583/1 "2017-12-06T23:38:32Z")

</div>

Hi,

I am getting slow performance when using logstash-filter-elasticsearch. Even with small volumes of data. I've tried this in multiple versions from 5.3 up to 6.0 of both ES & LS.

By way of example here is a config file:

```
input {
        jdbc {
        jdbc_connection_string => "jdbc:mysql://mydb"
                jdbc_user => "myuser"
                jdbc_password => "mypassword"
                jdbc_driver_library => "mysql-connector-java-5.1.42-bin.jar"
                jdbc_driver_class => "com.mysql.jdbc.Driver"
                    statement =>

		"SELECT * FROM table where id > :sql_last_value limit 250"

		tracking_column => id
		use_column_value => true
		record_last_run => true
		last_run_metadata_path => "views.txt"
    }
}

filter{
  elasticsearch {
    hosts => "https://URL:9243" 
    index => "reference"
    user => "elastic"
    password => "password"
    query => "product_code:%{[media_id]}"
    fields => [["aaa","aaa"]]
  }

filter {
           date {
                 match => ["purchase_time","YYYY-MM-dd HH:mm:ss"]
                target => "@timestamp"
           }
}

output {
  elasticsearch {
  hosts => "https://URL:9243"
  index => "myindex"
  user => "elastic"
  password => "password"
  document_type => "media"
  }

}

```

My ES Reference index only has a few thousand records & only 15 columns.

I was running on ES Cloud & Local Cluster (huge environments - nothing running) and observed slowness. Any ideas how i'd go about debugging.

For Local testing I ran this on my Mac Book Pro which Intel Core I7 2.7GHz 16GB Ram and SSD with macOS Sierra 10.12.6. Java Versions:

```
java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

```

For ES Cloud I tried this on 5.6.4 and 6.0

Thanks  
Wayne

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 18, 2017, 7:19pm UTC](https://discuss.elastic.co/t/logstash-filter-elasticsearch-slow/110583/2 "2017-12-18T19:19:05Z")

</div>

What does "slow" mean? How many events can you process each second?

---

<div class="post-metadata">

**Author:** ![Wayne\_Taylor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wayne_taylor/32/45984_2.png) [@Wayne\_Taylor](https://discuss.elastic.co/u/Wayne_Taylor)\
**Post date:** [December 18, 2017, 8:19pm UTC](https://discuss.elastic.co/t/logstash-filter-elasticsearch-slow/110583/3 "2017-12-18T20:19:00Z")

</div>

Slow defined as 500 documents ingest in a few minutes when it can load hundreds of thousands without it in a minute

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 19, 2017, 6:25am UTC](https://discuss.elastic.co/t/logstash-filter-elasticsearch-slow/110583/4 "2017-12-19T06:25:58Z")

</div>

That is slow, yes, but keep in mind that each event leads to an ES query that easily requires an order of milliseconds so it'll never be terribly high-performant. I'd look into the performance metrics of ES and Logstash to what's taking so much time. Is it ES itself or does Logstash add overhead? Also, have you tried raising the number of pipeline workers? Since your elasticsearch filter add latency from an external resource I'd expect Logstash to be idling a lot of the time so you should be able to raise the concurrency quite a bit.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [December 19, 2017, 7:57am UTC](https://discuss.elastic.co/t/logstash-filter-elasticsearch-slow/110583/5 "2017-12-19T07:57:11Z")

</div>

It looks like you are querying a cloud instance from your laptop. Have you tried running Logstash closer to Elasticsearch to minimise latency?

---

<div class="post-metadata">

**Author:** ![Wayne\_Taylor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wayne_taylor/32/45984_2.png) [@Wayne\_Taylor](https://discuss.elastic.co/u/Wayne_Taylor)\
**Post date:** [December 22, 2017, 1:10am UTC](https://discuss.elastic.co/t/logstash-filter-elasticsearch-slow/110583/6 "2017-12-22T01:10:31Z")

</div>

Hi @magnusbaeck , increasing workers is a little faster yes . I'll add in some metrics per request.

@Christian_Dahlqvist - I've tried on a local ES cluster too and same issue. Thanks for responding though.

Wayne

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 19, 2018, 1:10am UTC](https://discuss.elastic.co/t/logstash-filter-elasticsearch-slow/110583/7 "2018-01-19T01:10:39Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.