[kibana] Socket hang up, unable to revive connection

[soulou]

I'm opening this ticket as a followup of this one.

Socket hang up kibana

@Brandon_Kobel Yes, we do have a load balancer in-front of Elasticsearch. We can navigate each page in kibana but kibana is not showing any results. Every time we need to restart kibana and when we restart it, it works fine utill another socket hang up error occurs.

I'm experiencing exactly the same behavior.

• At some point connections failed (socket hangup, unable to revive connection, no living connection)
• Restarting kibana makes it work again.
• Elasticsearch 3 nodes cluster is all green no special logs in it.

Yes there is a HAProxy in front of my Elasticsearch cluster, with an idle timeout of 1h. I assume this is the problem here.

Is there a way to configure kibana via elasticsearch.js to handle this properly?

Regards,

-- Léo

[Larry_Gregory]

@soulou

Yes there is a HAProxy in front of my Elasticsearch cluster, with an idle timeout of 1h. I assume this is the problem here.

Can you clarify what "idle timeout" is? I'm not terribly familiar with HAProxy.

What is your "timeout server" set to? I believe you'll want to make sure this is set to something at least as large as your elasticsearch.requestTimeout setting within kibana.yml. This defaults to 30000 (30 seconds), so we'll want to ensure that HAProxy allows Elasticsearch at least that long to respond to requests.

[soulou]

Hi Larry Gregory, thanks a lot for your answer.

Sorry for using idle timeout it wasn't accurate and plus, the value was wrong: the current configuration is:

    timeout connect 10s
    timeout client 1m
    timeout server 2m
    timeout check 5s
    timeout http-request 5s

So the timeout server is 2 minutes, while requestTimeout of kibana is at 30 seconds.

But I had this feeling that then connection is not close but put back in a pool, and a bit later kibana reused it but haproxy closed it as nothing went through the pipe. Does it sound possible to you?

Regards,

[Larry_Gregory]

Thanks for the clarification.

@soulou

timeout http-request 5s

This one looks suspicious to me...can you increase this to 2m and see if that helps?

[soulou]

Ok we're going to try this and I'll come back to you. From what I got the http-request timeout aims at ensuring no client open connections but don't send any request data. But maybe it leads to some issues with http keepalive elasticsearch.js seems to use.