# Socket hangs up while running Timelion

**URL:** <https://discuss.elastic.co/t/socket-hangs-up-while-running-timelion/97396>\
**Category:** Elasticsearch\
**Created:** [August 17, 2017, 11:19am UTC](https://discuss.elastic.co/t/socket-hangs-up-while-running-timelion/97396 "2017-08-17T11:19:18Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![sreejithps303](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sreejithps303/32/23868_2.png) [@sreejithps303](https://discuss.elastic.co/u/sreejithps303)\
**Post date:** [August 17, 2017, 11:19am UTC](https://discuss.elastic.co/t/socket-hangs-up-while-running-timelion/97396/1 "2017-08-17T11:19:18Z")

</div>

Hi ,

I am getting socket hangs up error in kibana while running timelion visualisations.Elasticsearch load also increasing drastically.  
I have tried the below things. set thread\_pool.search.size: 20 and thread\_pool.search.queue\_size: 20000 in elasticsearch . Also http.max\_header\_size: 16kb.

But these did not helped. there is no issues with other visualisations or dashboards .  
Can any one helps with the possible causes.  
Elastic stack version 5.5.0

---

<div class="post-metadata">

**Author:** ![cjcenizal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cjcenizal/32/11216_2.png) [@cjcenizal](https://discuss.elastic.co/u/cjcenizal)\
**Post date:** [August 21, 2017, 7:26pm UTC](https://discuss.elastic.co/t/socket-hangs-up-while-running-timelion/97396/2 "2017-08-21T19:26:29Z")

</div>

Hi there, take a look at [Elasticsearch timeouts and Kibana Discover: socket hang up](https://discuss.elastic.co/t/elasticsearch-timeouts-and-kibana-discover-socket-hang-up/66378). In that thread, one solution was to increase the elasticsearch.requestTimeout setting in Kibana: [https://www.elastic.co/guide/en/kibana/5.0/settings.html](https://www.elastic.co/guide/en/kibana/5.0/settings.html). Does this help?

Thanks,  
CJ

---

<div class="post-metadata">

**Author:** ![cjcenizal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cjcenizal/32/11216_2.png) [@cjcenizal](https://discuss.elastic.co/u/cjcenizal)\
**Post date:** [August 21, 2017, 7:27pm UTC](https://discuss.elastic.co/t/socket-hangs-up-while-running-timelion/97396/3 "2017-08-21T19:27:31Z")

</div>

If that doesn't work, here's a similar thread where the user solved the problem with different elasticsearch.yml settings: ["socket hang up"](https://discuss.elastic.co/t/socket-hang-up/87763/5).

---

<div class="post-metadata">

**Author:** ![sreejithps303](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sreejithps303/32/23868_2.png) [@sreejithps303](https://discuss.elastic.co/u/sreejithps303)\
**Post date:** [August 22, 2017, 2:37am UTC](https://discuss.elastic.co/t/socket-hangs-up-while-running-timelion/97396/4 "2017-08-22T02:37:50Z")

</div>

Thanks for replying!  
I had tried both these options. no luck. Elasticsearch request timeout i set to a very high value.  
`elasticsearch.requestTimeout: 100000000`  
Also tried the http settings as well. but issue still persists. I have only issues with Timelion queries,visualisations . Normal searches visualizations and dashboards are working fine.Even searched with long periods doesnt have any issue.  
Below are the logs i could see in kibana.

```auto
{  
   "type":"ops",
   "@timestamp":"2017-08-22T02:24:30Z",
   "tags":[  

   ],
   "pid":20469,
   "os":{  
      "load":[  
         13.2158203125,
         6.29736328125,
         3.83642578125
      ],
      "mem":{  
         "total":25113096192,
         "free":203698176
      },
      "uptime":1385344
   },
   "proc":{  
      "uptime":987.707,
      "mem":{  
         "rss":87363584,
         "heapTotal":66367488,
         "heapUsed":55651992,
         "externa l":454281
      },
      "delay":0.4007890224456787
   },
   "load":{  
      "requests":{  
         "80":{  
            "total":0,
            "disconnects":0,
            "statusCodes":{  

            }
         }
      },
      "concurrents":{  
         "80":6
      },
      "responseTi mes":{  
         "80":{  
            "avg":null,
            "max":0
         }
      },
      "sockets":{  
         "http":{  
            "total":0
         },
         "https":{  
            "total":0
         }
      }
   },
   "message":"memory: 53.1MB uptime: 0:16:28 load: [13.22 6.30 3.84] delay: 0.401"
}{  
   "type":"log",
   "@timestamp":"2017-08-22T02:24:34Z",
   "tags":[  
      "error",
      "elasticsearch",
      "data"
   ],
   "pid":20469,
   "message":"Request complete with error\nPO ST http://xx.xx.xx.xx:9200/_all/_search => socket hang up"
}{  
   "type":"log",
   "@timestamp":"2017-08-22T02:24:34Z",
   "tags":[  
      "error",
      "elasticsearch",
      "data"
   ],
   "pid":20469,
   "message":"Request complete with error\nPO ST http://xx.xx.xx.xx:9200/_all/_search => socket hang up"
}{  
   "type":"log",
   "@timestamp":"2017-08-22T02:24:34Z",
   "tags":[  
      "error",
      "elasticsearch",
      "data"
   ],
   "pid":20469,
   "message":"Request complete with error\nPO ST http://xx.xx.xx.xx:9200/_all/_search => socket hang up"
}{  
   "type":"log",
   "@timestamp":"2017-08-22T02:24:34Z",
   "tags":[  
      "error",
      "elasticsearch",
      "data"
   ],
   "pid":20469,
   "message":"Request complete with error\nPO ST http://xx.xx.xx.xx:9200/_all/_search => socket hang up"
}

```

---

<div class="post-metadata">

**Author:** ![cjcenizal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cjcenizal/32/11216_2.png) [@cjcenizal](https://discuss.elastic.co/u/cjcenizal)\
**Post date:** [August 22, 2017, 5:13pm UTC](https://discuss.elastic.co/t/socket-hangs-up-while-running-timelion/97396/5 "2017-08-22T17:13:06Z")

</div>

Hmm, odd. Could you share the Timelion query and date range you're using?

Thanks,  
CJ

---

<div class="post-metadata">

**Author:** ![sreejithps303](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sreejithps303/32/23868_2.png) [@sreejithps303](https://discuss.elastic.co/u/sreejithps303)\
**Post date:** [August 22, 2017, 5:38pm UTC](https://discuss.elastic.co/t/socket-hangs-up-while-running-timelion/97396/6 "2017-08-22T17:38:16Z")

</div>

Any query other than `.es(*)` hangs. Even for time range 15 mins. I have tried to import one day prod index to non production environment and tried to run the same Timelion queries. It is working fine there.  
We have searchgurd installed , will that have any impact? ( It is there in Non prod also , but there i dont see any problem)

---

<div class="post-metadata">

**Author:** ![sreejithps303](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sreejithps303/32/23868_2.png) [@sreejithps303](https://discuss.elastic.co/u/sreejithps303)\
**Post date:** [September 1, 2017, 9:31am UTC](https://discuss.elastic.co/t/socket-hangs-up-while-running-timelion/97396/7 "2017-09-01T09:31:04Z")

</div>

Hi,

Anyone else having slimier kind of issues? any ES tuning suggestions?

---

<div class="post-metadata">

**Author:** ![sreejithps303](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sreejithps303/32/23868_2.png) [@sreejithps303](https://discuss.elastic.co/u/sreejithps303)\
**Post date:** [September 1, 2017, 9:56am UTC](https://discuss.elastic.co/t/socket-hangs-up-while-running-timelion/97396/8 "2017-09-01T09:56:40Z")

</div>

I could see this error in elasticsearch logs

````auto
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.transport.TransportSer
vice$7@6b6febc0 on EsThreadPoolExecutor[search, queue capacity = 80000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@458097ce[R
unning, pool size = 10, active threads = 10, queued tasks = 80000, completed tasks = 485826]]
	at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:50) ~[elasticsearch-5.5.0.jar:5.5.0]
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_144]
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) ~[?:1.8.0_144]
	at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.doExecute(EsThreadPoolExecutor.java:94) ~[elasticsearch-5.5.0.jar:5.5.0
]
	at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:89) ~[elasticsearch-5.5.0.jar:5.5.0]
	at org.elasticsearch.transport.TransportService.sendLocalRequest(TransportService.java:640) ~[elasticsearch-5.5.0.jar:5.5.0]
	at org.elasticsearch.transport.TransportService.access$000(TransportService.java:74) ~[elasticsearch-5.5.0.jar:5.5.0]
	at org.elasticsearch.transport.TransportService$3.sendRequest(TransportService.java:137) ~[elasticsearch-5.5.0.jar:5.5.0]
	at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:586) ~[elasticsearch-5.5.0.jar:5.5.0]
	at com.floragunn.searchguard.transport.SearchGuardInterceptor.sendRequestDecorate(SearchGuardInterceptor.java:93) ~[?:?]
	at com.floragunn.searchguard.SearchGuardPlugin$2$2.sendRequest(SearchGuardPlugin.java:355) ~[?:?]
	at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:519) ~[elasticsearch-5.5.0.jar:5.5.0]
	at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:546) ~[elasticsearch-5.5.0.jar:5.5.0]
	at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:537) ~[elasticsearch-5.5.0.jar:5.5.0]
	at org.elasticsearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:137) ~[elasticsearch-5.5.0.jar:5.
5.0]
```
````

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 29, 2017, 9:56am UTC](https://discuss.elastic.co/t/socket-hangs-up-while-running-timelion/97396/9 "2017-09-29T09:56:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
