# Logstash filter elasticsearch open connections issue **URL:** \ **Category:** Logstash\ **Created:** [October 6, 2017, 11:35pm UTC](https://discuss.elastic.co/t/logstash-filter-elasticsearch-open-connections-issue/103091 "2017-10-06T23:35:16Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![MaheshP](https://avatars.discourse-cdn.com/v4/letter/m/6de8d8/32.png) [@MaheshP](https://discuss.elastic.co/u/MaheshP)\ **Post date:** [October 6, 2017, 11:35pm UTC](https://discuss.elastic.co/t/logstash-filter-elasticsearch-open-connections-issue/103091/1 "2017-10-06T23:35:16Z")
Hello, Currently we are trying to send data from kafka to elasticsearch and in between we used logstash filter elasticsearch plugin to query for previous documents timestamp and other fields, at that time open connections between logstash and elasticsearch are very high. Due to open connections some threads are waiting and then logstash was crashed. **Is there any way to reduce the open connections between logstash and elasticsearch in logstash filter elasticsearch plugin?** **Ref:** **Kafka QPS(Queries per second):** 50000 **Conf in logstash filter elasticsearch plugin:** input { kafka { } } filter { elasticsearch { hosts =\> ["localhost:9200"] index =\> "test\_index\*" enable\_sort =\> false query =\> "_type:type_%{[oi]} AND \_id:%{[d]}-%{[ai]}" fields =\> [["ts","stime"],["ml","mlarray"],["dl","dlarray"],["wl","wlarray"]] } } output { elasticsearch { } } Please help me. ---
**Author:** ![MaheshP](https://avatars.discourse-cdn.com/v4/letter/m/6de8d8/32.png) [@MaheshP](https://discuss.elastic.co/u/MaheshP)\ **Post date:** [October 7, 2017, 1:56pm UTC](https://discuss.elastic.co/t/logstash-filter-elasticsearch-open-connections-issue/103091/2 "2017-10-07T13:56:39Z")
Hi Elastic team, Could you help us for the above issue? or any advise would be nice. **Is there any way to reduce the open connections between logstash and elasticsearch in logstash filter elasticsearch plugin?** ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 7, 2017, 2:56pm UTC](https://discuss.elastic.co/t/logstash-filter-elasticsearch-open-connections-issue/103091/3 "2017-10-07T14:56:13Z")
This forum is manned by volunteers, so please be patient. ---
**Author:** ![MaheshP](https://avatars.discourse-cdn.com/v4/letter/m/6de8d8/32.png) [@MaheshP](https://discuss.elastic.co/u/MaheshP)\ **Post date:** [October 7, 2017, 3:57pm UTC](https://discuss.elastic.co/t/logstash-filter-elasticsearch-open-connections-issue/103091/4 "2017-10-07T15:57:23Z")
Ok thank you @Christian_Dahlqvist . I look forward to hearing good solution/advice from your team. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 7, 2017, 5:20pm UTC](https://discuss.elastic.co/t/logstash-filter-elasticsearch-open-connections-issue/103091/5 "2017-10-07T17:20:30Z")
Which version of Logstash are you using? How many open connections are you seeing? How many worker threads is Logstash using? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 7, 2017, 7:01pm UTC](https://discuss.elastic.co/t/logstash-filter-elasticsearch-open-connections-issue/103091/6 "2017-10-07T19:01:35Z")
> [@MaheshP](#): > > Due to open connections some threads are waiting and then logstash was crashed. Please show the logs for the time that this issue happened. ---
**Author:** ![MaheshP](https://avatars.discourse-cdn.com/v4/letter/m/6de8d8/32.png) [@MaheshP](https://discuss.elastic.co/u/MaheshP)\ **Post date:** [October 7, 2017, 9:42pm UTC](https://discuss.elastic.co/t/logstash-filter-elasticsearch-open-connections-issue/103091/7 "2017-10-07T21:42:27Z")
@Christian_Dahlqvist Thank you for quick response. Please find the details. Which version of Logstash are you using? > **logstash 5.4.3** How many open connections are you seeing? > **(~17000 Connections)** How many worker threads is Logstash using? > **Configurations are default. We did POC on 2 core machine and the pipeline workers by default "2".** **Problem:** Open connections between logstash and elasticsearch are high, while querying for previous documents fields by using **logstash filter elasticsearch plugin** in **logstash**. And also there are so many **TIME\_WAIT** connections on logstash server. **Note:** No issue when we send events directly from kafka to elasticsearch by using logstash without elasticsearch filter plugin. ---
**Author:** ![MaheshP](https://avatars.discourse-cdn.com/v4/letter/m/6de8d8/32.png) [@MaheshP](https://discuss.elastic.co/u/MaheshP)\ **Post date:** [October 7, 2017, 9:46pm UTC](https://discuss.elastic.co/t/logstash-filter-elasticsearch-open-connections-issue/103091/8 "2017-10-07T21:46:39Z")
@warkolm there are no errors and warnings in logstash logs > [2017-09-19T02:59:55,993][INFO][logstash.pipeline] Pipeline main started > [2017-09-19T02:59:56,095][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600} ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 7, 2017, 9:48pm UTC](https://discuss.elastic.co/t/logstash-filter-elasticsearch-open-connections-issue/103091/9 "2017-10-07T21:48:47Z")
> [@MaheshP](#): > > there are no errors and warnings in logstash logs Then how do you know it crashed? ---
**Author:** ![MaheshP](https://avatars.discourse-cdn.com/v4/letter/m/6de8d8/32.png) [@MaheshP](https://discuss.elastic.co/u/MaheshP)\ **Post date:** [October 7, 2017, 9:55pm UTC](https://discuss.elastic.co/t/logstash-filter-elasticsearch-open-connections-issue/103091/10 "2017-10-07T21:55:57Z")
@warkolm Kafka consumer lag is increasing for that logstash consumer, and when I checked consumer threads by using below command it was not showing any consumer logstash threads and it's not consuming any events. > bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group ${LOGSTASH\_GROUP} ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 4, 2017, 10:10pm UTC](https://discuss.elastic.co/t/logstash-filter-elasticsearch-open-connections-issue/103091/11 "2017-11-04T22:10:31Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.