Socket hangs up while running Timelion

Elastic Stack Elasticsearch
1

Hi ,

I am getting socket hangs up error in kibana while running timelion visualisations.Elasticsearch load also increasing drastically.
I have tried the below things. set thread_pool.search.size: 20 and thread_pool.search.queue_size: 20000 in elasticsearch . Also http.max_header_size: 16kb.

But these did not helped. there is no issues with other visualisations or dashboards .
Can any one helps with the possible causes.
Elastic stack version 5.5.0

2

Hi there, take a look at Elasticsearch timeouts and Kibana Discover: socket hang up. In that thread, one solution was to increase the elasticsearch.requestTimeout setting in Kibana: https://www.elastic.co/guide/en/kibana/5.0/settings.html. Does this help?

Thanks,
CJ

3

If that doesn't work, here's a similar thread where the user solved the problem with different elasticsearch.yml settings: "socket hang up".

4

Thanks for replying!
I had tried both these options. no luck. Elasticsearch request timeout i set to a very high value.
elasticsearch.requestTimeout: 100000000
Also tried the http settings as well. but issue still persists. I have only issues with Timelion queries,visualisations . Normal searches visualizations and dashboards are working fine.Even searched with long periods doesnt have any issue.
Below are the logs i could see in kibana.

{  
   "type":"ops",
   "@timestamp":"2017-08-22T02:24:30Z",
   "tags":[  

   ],
   "pid":20469,
   "os":{  
      "load":[  
         13.2158203125,
         6.29736328125,
         3.83642578125
      ],
      "mem":{  
         "total":25113096192,
         "free":203698176
      },
      "uptime":1385344
   },
   "proc":{  
      "uptime":987.707,
      "mem":{  
         "rss":87363584,
         "heapTotal":66367488,
         "heapUsed":55651992,
         "externa l":454281
      },
      "delay":0.4007890224456787
   },
   "load":{  
      "requests":{  
         "80":{  
            "total":0,
            "disconnects":0,
            "statusCodes":{  

            }
         }
      },
      "concurrents":{  
         "80":6
      },
      "responseTi mes":{  
         "80":{  
            "avg":null,
            "max":0
         }
      },
      "sockets":{  
         "http":{  
            "total":0
         },
         "https":{  
            "total":0
         }
      }
   },
   "message":"memory: 53.1MB uptime: 0:16:28 load: [13.22 6.30 3.84] delay: 0.401"
}{  
   "type":"log",
   "@timestamp":"2017-08-22T02:24:34Z",
   "tags":[  
      "error",
      "elasticsearch",
      "data"
   ],
   "pid":20469,
   "message":"Request complete with error\nPO ST http://xx.xx.xx.xx:9200/_all/_search => socket hang up"
}{  
   "type":"log",
   "@timestamp":"2017-08-22T02:24:34Z",
   "tags":[  
      "error",
      "elasticsearch",
      "data"
   ],
   "pid":20469,
   "message":"Request complete with error\nPO ST http://xx.xx.xx.xx:9200/_all/_search => socket hang up"
}{  
   "type":"log",
   "@timestamp":"2017-08-22T02:24:34Z",
   "tags":[  
      "error",
      "elasticsearch",
      "data"
   ],
   "pid":20469,
   "message":"Request complete with error\nPO ST http://xx.xx.xx.xx:9200/_all/_search => socket hang up"
}{  
   "type":"log",
   "@timestamp":"2017-08-22T02:24:34Z",
   "tags":[  
      "error",
      "elasticsearch",
      "data"
   ],
   "pid":20469,
   "message":"Request complete with error\nPO ST http://xx.xx.xx.xx:9200/_all/_search => socket hang up"
}
5

Hmm, odd. Could you share the Timelion query and date range you're using?

Thanks,
CJ

6

Any query other than .es(*) hangs. Even for time range 15 mins. I have tried to import one day prod index to non production environment and tried to run the same Timelion queries. It is working fine there.
We have searchgurd installed , will that have any impact? ( It is there in Non prod also , but there i dont see any problem)

7

Hi,

Anyone else having slimier kind of issues? any ES tuning suggestions?

8

I could see this error in elasticsearch logs

Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.transport.TransportSer
vice$7@6b6febc0 on EsThreadPoolExecutor[search, queue capacity = 80000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@458097ce[R
unning, pool size = 10, active threads = 10, queued tasks = 80000, completed tasks = 485826]]
	at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:50) ~[elasticsearch-5.5.0.jar:5.5.0]
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_144]
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) ~[?:1.8.0_144]
	at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.doExecute(EsThreadPoolExecutor.java:94) ~[elasticsearch-5.5.0.jar:5.5.0
]
	at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:89) ~[elasticsearch-5.5.0.jar:5.5.0]
	at org.elasticsearch.transport.TransportService.sendLocalRequest(TransportService.java:640) ~[elasticsearch-5.5.0.jar:5.5.0]
	at org.elasticsearch.transport.TransportService.access$000(TransportService.java:74) ~[elasticsearch-5.5.0.jar:5.5.0]
	at org.elasticsearch.transport.TransportService$3.sendRequest(TransportService.java:137) ~[elasticsearch-5.5.0.jar:5.5.0]
	at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:586) ~[elasticsearch-5.5.0.jar:5.5.0]
	at com.floragunn.searchguard.transport.SearchGuardInterceptor.sendRequestDecorate(SearchGuardInterceptor.java:93) ~[?:?]
	at com.floragunn.searchguard.SearchGuardPlugin$2$2.sendRequest(SearchGuardPlugin.java:355) ~[?:?]
	at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:519) ~[elasticsearch-5.5.0.jar:5.5.0]
	at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:546) ~[elasticsearch-5.5.0.jar:5.5.0]
	at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:537) ~[elasticsearch-5.5.0.jar:5.5.0]
	at org.elasticsearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:137) ~[elasticsearch-5.5.0.jar:5.
5.0]
```
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.