120s Timeout on Elasticsearch Queries via Kibana

Elastic Stack Kibana
1

I have two clusters (ES 7.3.1, bundled OpenJDK 12) that are hosted by an internal team here at my org. Each cluster has 6 data nodes, three master nodes, and one cluster has two client nodes that perform searches against both clusters simultaneously.

We are consistently hitting a 2 minute timeout on all queries ran through Kibana that is showing up as a 502 Bad Gateway, the Kibana logs show a socket hang up at the same time.

The team hosting these clusters is now saying that the two-minute timeout is due to a setting (unexposed prior to 7.4) named server.keepalivetimeout seen in the code here.

The issue is: I'd expect a timeout in Kibana to be handled more gracefully than being masked in the UI as a 502 gateway timeout, and I'm pretty sure that in the past I've successfully ran >2minute queries on older-versioned clusters.

Can anyone provide any clarification on whether I should be expecting to hit a hardcoded timeout at exactly 2 minutes?

Edit: we've made some modifications to the cluster and now the error at 2 minutes is showing up as:
image

2

Hey @seth.yes,

Hmm, I'm just as surprised as you are about the 502 gateway timeout error. Do you happen to have a proxy setup in-between Kibana and Elasticsearch?

When did these errors start happening? Was this following an upgrade, was it always this way, or did it just start happening one day for no apparent reason?

3

Hi @Larry_Gregory , yes we do have a proxy setup between Elasticsearch and Kibana. The supporting team believes the timeout is occurring natively to Kibana.

I just want to verify that we should typically be able to run >2m queries in Kibana 7.3, is that correct?

This is a new cluster deployed on 7.3, >2m queries have never worked in this new cluster.

4

I was able to confirm that your operations folks are correct -- the 2 minute timeout is a hard limit in version 7.3, but 7.4 and beyond can configure this via server.socketTimeout and server.keepAliveTimeout

1 Like
5

@Larry_Gregory thanks for the conformation. Looks like the latest downloadble version is still 7.3.2 as shown in https://www.elastic.co/downloads/kibana. When 7.4 will be open to download? We're looking forward to trying it out as soon as possible.

1 Like
6

We don't have a fixed release date, but we're working hard on making it available as soon as possible.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.