Kibana Timeout on visualization edit

[ramaro0x00 (Goncalo Amaro)]

Hello guys,

I'm having troubles when creating, updating and deleting dashboards and/or visualizations. When opening the same dashboards and visualizations I have no problem whatsoever. I have a nearly empty database. This problem appeared after updating from 6.1 to 6.2.4.

After changing timeout 30000 in Kibana, I started getting instead a 504 Response socket hang up:

{"type":"log","@timestamp":"2018-06-04T17:54:03Z","tags":["error","elasticsearch","admin"],"pid":30905,"message":"Request error, retrying\nPOST http://localhost:9200/.kibana/doc/visualization%3AEvents_Clients-slash-Hour?refresh=wait_for => socket hang up"}

Have nothing in Elasticsearch log. Did anybody went through the same?

[bhavyarm]

Hi,

Did you update your ES to 6.2.4 too? Can you please check that?
Also whats the output of this: http://localhost:9200/.kibana

Thanks,
Bhavya

[ramaro0x00 (Goncalo Amaro)]

Hi,

Thanks for your feedback. I've updated through normal apt-get upgrade. It does update all stack, correct?

Bellow the output:

curl -XGET 'localhost:9200/.kibana' {".kibana":{"aliases":{},"mappings":{"doc":{"dynamic":"strict","properties":{"config":{"dynamic":"true","properties":{"buildNum":{"type":"keyword"},"defaultIndex":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"dashboard":{"properties":{"description":{"type":"text"},"hits":{"type":"integer"},"kibanaSavedObjectMeta":{"properties":{"searchSourceJSON":{"type":"text"}}},"optionsJSON":{"type":"text"},"panelsJSON":{"type":"text"},"refreshInterval":{"properties":{"display":{"type":"keyword"},"pause":{"type":"boolean"},"section":{"type":"integer"},"value":{"type":"integer"}}},"timeFrom":{"type":"keyword"},"timeRestore":{"type":"boolean"},"timeTo":{"type":"keyword"},"title":{"type":"text"},"uiStateJSON":{"type":"text"},"version":{"type":"integer"}}},"graph-workspace":{"properties":{"description":{"type":"text"},"kibanaSavedObjectMeta":{"properties":{"searchSourceJSON":{"type":"text"}}},"numLinks":{"type":"integer"},"numVertices":{"type":"integer"},"title":{"type":"text"},"version":{"type":"integer"},"wsState":{"type":"text"}}},"index-pattern":{"properties":{"fieldFormatMap":{"type":"text"},"fields":{"type":"text"},"intervalName":{"type":"keyword"},"notExpandable":{"type":"boolean"},"sourceFilters":{"type":"text"},"timeFieldName":{"type":"keyword"},"title":{"type":"text"}}},"search":{"properties":{"columns":{"type":"keyword"},"description":{"type":"text"},"hits":{"type":"integer"},"kibanaSavedObjectMeta":{"properties":{"searchSourceJSON":{"type":"text"}}},"sort":{"type":"keyword"},"title":{"type":"text"},"version":{"type":"integer"}}},"server":{"properties":{"uuid":{"type":"keyword"}}},"timelion-sheet":{"properties":{"description":{"type":"text"},"hits":{"type":"integer"},"kibanaSavedObjectMeta":{"properties":{"searchSourceJSON":{"type":"text"}}},"timelion_chart_height":{"type":"integer"},"timelion_columns":{"type":"integer"},"timelion_interval":{"type":"keyword"},"timelion_other_interval":{"type":"keyword"},"timelion_rows":{"type":"integer"},"timelion_sheet":{"type":"text"},"title":{"type":"text"},"version":{"type":"integer"}}},"type":{"type":"keyword"},"updated_at":{"type":"date"},"url":{"properties":{"accessCount":{"type":"long"},"accessDate":{"type":"date"},"createDate":{"type":"date"},"url":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":2048}}}}},"visualization":{"properties":{"description":{"type":"text"},"kibanaSavedObjectMeta":{"properties":{"searchSourceJSON":{"type":"text"}}},"savedSearchId":{"type":"keyword"},"title":{"type":"text"},"uiStateJSON":{"type":"text"},"version":{"type":"integer"},"visState":{"type":"text"}}}}}},"settings":{"index":{"refresh_interval":"-1","number_of_shards":"1","provided_name":".kibana","creation_date":"1510847841851","number_of_replicas":"0","uuid":"T8uTFtgYQ7KJ0eW9yrs_UA","version":{"created":"6000099","upgraded":"6020499"}}}}}

One thing I question myself about is the elasticsearch jvm.options file. Not sure if I didn't went wrong keeping previous file on update, even doing diffs with the new configurations. Can it be anyhow related to it?

Thanks again,
Gonçalo

[bhavyarm]

HI,

Can you reach elasticsearch through your Kibana?
Query your cluster data through devtools and console in Kibana?

Thanks,
Bhavya

[ramaro0x00 (Goncalo Amaro)]

Hi,

Yes, I'm attaching an example, to see if this is what you meant. Sorry for the blank square in the middle for anonymization.

Despite the timeout or 504 errors, all visualizations and dashboards I try to edit, get created after a while (even though it takes long) and when accessing them, all works perfectly.

I've detected this because the users were clicking frenetically over the save button, due to the fact that kibana didn't respond to those changes submissions. After 5 minutes I had 10 times the same visualization created.

This is a simple setup, a fresh install is an option, if nothing else comes to your mind. Let me know if it does, though.

Thanks again for your time and availability.

[bhavyarm]

@timroes can you please take a look at this?

Thanks,
Bhavya

[timroes]

Looks more like an issue with the Kibana/ES connection rather than anything visualization/dashboard specific. Maybe @tylersmalley could help with that one?

[ramaro0x00 (Goncalo Amaro)]

@tylersmalley, did you have the opportunity to have a look at this?

Thanks!
Gonçalo

[tylersmalley]

@ramaro0x00, can you describe your setup? Is Kibana and ES on separate machines? What is the networking look like between the two? Is there a proxy between ES and Kibana?

[ramaro0x00 (Goncalo Amaro)]

Hi,

Thanks in advance.

Logstash is in different machine(s), sending logs to Elastic.

Elastic and Kibana are on the same machine, so no proxies or firewalls in between.

I find odd that it can be related to this communication, since I perfectly can read&search&filter logs, visualizations and dashboards. The problem only kicks off, when I try to store new/updated visualizations/dashboards.

BR,
Gonçalo

[tylersmalley]

With ES on the same box as Kibana, you really shouldn't have any connection issues and definitely shouldn't need a 30 second timeout.

The only thing I can think of is your ES instance is under provisioned. Can I get _cat/indices and _cluster/health?

[ramaro0x00 (Goncalo Amaro)]

Hi,

Please find it in attachment. Thank you.

BR,
Gonçalo Amaro

[tsullivan (Tim Sullivan)]

Hi, how are the memory resources looking for your ES nodes? I'd be curious to see what your JVM Heap, GC Duration, Index Memory and Read Threads charts in Monitoring look like. (Elasticsearch > Nodes > Node Detail > Advanced tab)

I'm having troubles when creating, updating and deleting dashboards and/or visualizations

When opening the same dashboards and visualizations I have no problem whatsoever

I'm not entirely sure, but it's possible that you're seeing this behavior from the benefit of request caching from Elasticsearch. Request caching can potentially return aggregation results instantly: Shard request cache settings | Elasticsearch Guide [8.11] | Elastic