# Logstash can't talk to elasticsearch after several hours' running **URL:** \ **Category:** Logstash\ **Created:** [October 17, 2016, 2:35pm UTC](https://discuss.elastic.co/t/logstash-cant-talk-to-elasticsearch-after-several-hours-running/63198 "2016-10-17T14:35:23Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![caoping](https://avatars.discourse-cdn.com/v4/letter/c/aca169/32.png) [@caoping](https://discuss.elastic.co/u/caoping)\ **Post date:** [October 17, 2016, 2:35pm UTC](https://discuss.elastic.co/t/logstash-cant-talk-to-elasticsearch-after-several-hours-running/63198/1 "2016-10-17T14:35:23Z")
I found that after several hours running, my logstash can't connect to elasticsearch any more. From logstash log, it said: > {:timestamp=\>"2016-10-17T04:39:40.312000+0200", :message=\>"Starting pipeline", :id=\>"main", :pipeline\_workers=\>8, :batch\_size=\>1000, :batch\_delay=\>5, :max\_inflight=\>8000, :level=\>:info} {:timestamp=\>"2016-10-17T04:39:40.325000+0200", :message=\>"Pipeline main started"} {:timestamp=\>"2016-10-17T06:18:55.520000+0200", :message=\>"Attempted to send a bulk request to Elasticsearch configured at '["[http://146.89.179.204:9200](http://146.89.179.204:9200)"]', but Elasticsearch appears to be unreachable or down!", :error\_message=\>"Connection timed out", :class=\>"Manticore::SocketException", :level=\>:error} However, I checked my elasticsearch was still running. What I'm using: - Filebeat: 1.2.3 - Logstash: 2.3.4 - Elasticsearch: 2.3.5 Any help greatly appreciated. Thanks a lot! ---
**Author:** ![caoping](https://avatars.discourse-cdn.com/v4/letter/c/aca169/32.png) [@caoping](https://discuss.elastic.co/u/caoping)\ **Post date:** [October 18, 2016, 9:43am UTC](https://discuss.elastic.co/t/logstash-cant-talk-to-elasticsearch-after-several-hours-running/63198/2 "2016-10-18T09:43:25Z")
Anyone met this issue before? Or anyone know what caused this kind of issue? I found some logs from logstash side. It said there were 3 running workers(8 workers in total), and each of these running workers were working on the job specified by the last plugin **date**. > {:timestamp=\>"2016-10-18T09:42:15.320000+0200", :message=\>#\58098, :events\_consumed=\>58098, :worker\_count=\>8, :inflight\_count=\>179, :**worker\_states=\>[{:status=\>"dead", :alive=\>false, :index=\>0, :inflight\_count=\>0}, {:status=\>"run", :alive=\>true, :index=\>1, :inflight\_count=\>36}, {:status=\>"dead", :alive=\>false, :index=\>2, :inflight\_count=\>0}, {:status=\>"dead", :alive=\>false, :index=\>3, :inflight\_count=\>0}, {:status=\>"dead", :alive=\>false, :index=\>4, :inflight\_count=\>0}, {:status=\>"run", :alive=\>true, :index=\>5, :inflight\_count=\>88}, {:status=\>"dead", :alive=\>false, :index=\>6, :inflight\_count=\>0}, {:status=\>"run", :alive=\>true, :index=\>7, :inflight\_count=\>55**}], :output\_info=\>[{:type=\>"elasticsearch", :config=\>{"hosts"=\>"146.89.179.204", "index"=\>"logstash-site-%{+YYYY.MM.dd}", "workers"=\>8, "flush\_size"=\>1000, " **ALLOW\_ENV**"=\>false}, :is\_multi\_worker=\>true, :events\_received=\>58098, :workers=\>\["146.89.179.204"], index=\>"logstash-site-%{+YYYY.MM.dd}", workers=\>8, flush\_size=\>1000, codec=\>\"UTF-8"\>, manage\_template=\>true, template\_name=\>"logstash", template\_overwrite=\>false, idle\_flush\_time=\>1, doc\_as\_upsert=\>false, max\_retries=\>3, script\_type=\>"inline", script\_var\_name=\>"event", scripted\_upsert=\>false, retry\_max\_interval=\>2, retry\_max\_items=\>500, retry\_on\_conflict=\>1, action=\>"index", ssl\_certificate\_verification=\>true, sniffing=\>false, sniffing\_delay=\>5\>, \["146.89.179.204"], index=\>"logstash-site-%{+YYYY.MM.dd}", workers=\>8, flush\_size=\>1000, codec=\>\"UTF-8"\>, manage\_template=\>true, template\_name=\>"logstash", template\_overwrite=\>false, idle\_flush\_time=\>1, doc\_as\_upsert=\>false, max\_retries=\>3, script\_type=\>"inline", script\_var\_name=\>"event", scripted\_upsert=\>false, retry\_max\_interval=\>2, retry\_max\_items=\>500, retry\_on\_conflict=\>1, action=\>"index", ssl\_certificate\_verification=\>true, sniffing=\>false, sniffing\_delay=\>5\>, \["146.89.179.204"], index=\>"logstash-site-%{+YYYY.MM.dd}", workers=\>8, flush\_size=\>1000, codec=\>\"UTF-8"\>, manage\_template=\>true, template\_name=\>"logstash", template\_overwrite=\>false, idle\_flush\_time=\>1, doc\_as\_upsert=\>false, max\_retries=\>3, script\_type=\>"inline", script\_var\_name=\>"event", scripted\_upsert=\>false, retry\_max\_interval=\>2, retry\_max\_items=\>500, retry\_on\_conflict=\>1, action=\>"index", ssl\_certificate\_verification=\>true, sniffing=\>false, sniffing\_delay=\>5\>, \["146.89.179.204"], index=\>"logstash-site-%{+YYYY.MM.dd}", workers=\>8, flush\_size=\>1000, codec=\>\"UTF-8"\>, manage\_template=\>true, template\_name=\>"logstash", template\_overwrite=\>false, idle\_flush\_time=\>1, doc\_as\_upsert=\>false, max\_retries=\>3, script\_type=\>"inline", script\_var\_name=\>"event", scripted\_upsert=\>false, retry\_max\_interval=\>2, retry\_max\_items=\>500, retry\_on\_conflict=\>1, action=\>"index", ssl\_certificate\_verification=\>true, sniffing=\>false, sniffing\_delay=\>5\>, this log line is too long, the rest please look at another response below. ---
**Author:** ![caoping](https://avatars.discourse-cdn.com/v4/letter/c/aca169/32.png) [@caoping](https://discuss.elastic.co/u/caoping)\ **Post date:** [October 18, 2016, 9:53am UTC](https://discuss.elastic.co/t/logstash-cant-talk-to-elasticsearch-after-several-hours-running/63198/3 "2016-10-18T09:53:52Z")
continued log: > :busy\_workers=\>3}], :thread\_info=\>[{"thread\_id"=\>28, "name"=\>"[main]\>worker1", "plugin"=\>["LogStash::Filters::Date", {"match"=\>["Logtime", "M/d/yy HH:mm:ss:SSS z"], "timezone"=\>"UTC"}], "backtrace"=\>["[...]/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:50:in `call'", "[...]/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:256:in `call\_once'", "[...]/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:153:in `code'", "[...]/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/http/manticore.rb:84:in `perform\_request'", "[...]/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/base.rb:257:in `call'", "[...]/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/base.rb:257:in `perform\_request'", "[...]/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/http/manticore.rb:67:in `perform_request'", "[...]/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/client.rb:128:in `perform\_request'", "[...]/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.0.18/lib/elasticsearch/api/actions/bulk.rb:90:in `bulk'", "[...]/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:53:in `non\_threadsafe\_bulk'", "[...]/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http\_client.rb:38:in `bulk'", "[...]/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in `synchronize'", "[...]/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http\_client.rb:38:in `bulk'", "[...]/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:172:in `safe\_bulk'", "[...]/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:101:in `submit'", "[...]/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:86:in `retrying\_submit'", "[...]/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:29:in `multi_receive'", "[...]/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:28:in `each\_slice'", "[...]/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:28:in `multi_receive'", "[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/output_delegator.rb:130:in `worker\_multi\_receive'", "[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/output\_delegator.rb:129:in `worker_multi_receive'", "[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/output_delegator.rb:114:in `multi\_receive'", "[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:301:in `output_batch'", "[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:301:in `each'", "[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:301:in `output_batch'", "[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:232:in `worker\_loop'", "[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:201:in `start_workers'"], "blocked_on"=>nil, "status"=>"run", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:50:in \`call'"}], :stalling\_threads\_info=\>[{"thread\_id"=\>28, "name"=\>"[main]\>worker1", "plugin"=\>["LogStash::Filters::Date", {"match"=\>["Logtime", "M/d/yy HH:mm:ss:SSS z"], "timezone"=\>"UTC"}], "current\_call"=\>"[...]/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:50:in `call'"}, {**"thread\_id"=\>32, "name"=\>"[main]\>worker5", "plugin"=\>["LogStash::Filters::Date", {"match"=\>["Logtime", "M/d/yy HH:mm:ss:SSS z"], "timezone"=\>"UTC"}], "current\_call"=\>"[...]/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:50:in `call'"}, {"thread_id"=\>34, "name"=\>"[main]\>worker7", "plugin"=\>["LogStash::Filters::Date", {"match"=\>["Logtime", "M/d/yy HH:mm:ss:SSS z"], "timezone"=\>"UTC"}], "current_call"=\>"[...]/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:50:in `call'"**}]}\>, :level=\>:warn} ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 18, 2016, 10:03am UTC](https://discuss.elastic.co/t/logstash-cant-talk-to-elasticsearch-after-several-hours-running/63198/4 "2016-10-18T10:03:27Z")
Is there anything in the Elasticsearch logs around the time the problems occurred? ---
**Author:** ![caoping](https://avatars.discourse-cdn.com/v4/letter/c/aca169/32.png) [@caoping](https://discuss.elastic.co/u/caoping)\ **Post date:** [October 21, 2016, 2:29am UTC](https://discuss.elastic.co/t/logstash-cant-talk-to-elasticsearch-after-several-hours-running/63198/5 "2016-10-21T02:29:58Z")
No. I didn't see anything in Elasticsearch logs. Every time I run into this issue, it can be resolved after I kill (kill -9) logstash, and start it again. I can't stop logstash safely. I have two VMs A and B, A is used as Logstash server, and B is both Logstash server and Elasticsearch server. The connection problem was only occurred in A. I'm not sure if this information helps to identify the error. These days, this issue came out very often, which has impacted our daily use greatly. Any help greatly appreciated. Thanks so much! ---
**Author:** ![Selle](https://avatars.discourse-cdn.com/v4/letter/s/a87d85/32.png) [@Selle](https://discuss.elastic.co/u/Selle)\ **Post date:** [October 21, 2016, 7:18pm UTC](https://discuss.elastic.co/t/logstash-cant-talk-to-elasticsearch-after-several-hours-running/63198/6 "2016-10-21T19:18:00Z")
Hi. Our logstash agents also had this issue. After a while they got the socket exception and they could never recover. Restarting them was the only way. The previous setup: Logstash 2.1 and 2.3 Elasticsearch 1.7 I ended up editing common.rb in the elasticsearch output to get any info about how many events it tries to send and how big. Often the size of the bulk request was small: 1 event and 500Bytes so probably not too big bulk request then 🙂 I ended up upgrading all agents to 2.4 and the error disappeared except from one node that sends around 200k events/hour. The good thing is that it can recover itself now. Hope it helps //Rickard ---
**Author:** ![caoping](https://avatars.discourse-cdn.com/v4/letter/c/aca169/32.png) [@caoping](https://discuss.elastic.co/u/caoping)\ **Post date:** [October 24, 2016, 3:03am UTC](https://discuss.elastic.co/t/logstash-cant-talk-to-elasticsearch-after-several-hours-running/63198/7 "2016-10-24T03:03:38Z")
Hi Rickard, Thanks so much for your information. I'll try the upgrade later to see if it can resolve my issue. Any progress will be updated here. Thanks again. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:33am UTC](https://discuss.elastic.co/t/logstash-cant-talk-to-elasticsearch-after-several-hours-running/63198/8 "2017-07-06T04:33:08Z")