Kibana dashboard weirdness

[adary (Yuval Adar)]

I have a cluster with either two or three nodes with elasticsearch, logstash and kibana.

ES is ok, logstash is ok, and kibana is ok as long as I don't try to programatically insert default index, visualisations and a dashboard.

First technical part.

I use:
ES 2.2.0
LS 2.2.2
Kibana 4.4.1

Kibana configuration is very straightforward: it connects to localhost:9200 on each node, on a higher level I have apache passthrough to allow authentication, and haproxy to load balance it all

If I don't fiddle with automatic definitions for default index, all is good.

The way I create the index is the following:

/usr/bin/curl -XPOST 'http://localhost:9200/.kibana/index-pattern/logstash-?op_type=create' -d '{"title":"logstash-","timeFieldName":"@timestamp"}'
/usr/bin/curl -XPOST 'http://localhost:9200/.kibana/config/4.4.1' -d '{"buildNum":9693, "defaultIndex":"logstash-*"}'

This is of course wrapped in a script that queries kibana for buildNum and version to form these URL's

This part seems to be fine since on it's own everything looks good, but as soon as I post the json files that define the dashboard and visualisation weirdness starts. I can open kibana and see the visualisations, but when I click on discover nothing happens (except some colors in the dashboard turning grey)

Injected json files are as following:

{
"hits": 0,
"timeRestore": false,
"description": "",
"title": "default",
"uiStateJSON": "{}",
"panelsJSON": "[{"col":1,"id":"Events-per-host","panelIndex":1,"row":4,"size_x":9,"size_y":5,"type":"visualization"},{"id":"Total-events","type":"visualization","panelIndex":2,"size_x":3,"size_y":2,"col":10,"row":4},{"id":"events-per-host-bar","type":"visualization","panelIndex":3,"size_x":12,"size_y":3,"col":1,"row":1}]",
"optionsJSON": "{"darkTheme":true}",
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{"filter":[{"query":{"query_string":{"analyze_wildcard":true,"query":"*"}}}]}"
}
}

{
"visState": "{"title":"New Visualization","type":"metric","params":{"fontSize":60},"aggs":[{"id":"1","type":"count","schema":"metric","params":{}}],"listeners":{}}",
"description": "",
"title": "Total events",
"uiStateJSON": "{}",
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{"index":"logstash-","query":{"query_string":{"query":"","analyze_wildcard":true}},"filter":[]}"
}
}

{
"visState": "{"title":"New Visualization","type":"pie","params":{"shareYAxis":true,"addTooltip":true,"addLegend":true,"isDonut":false},"aggs":[{"id":"1","type":"count","schema":"metric","params":{}},{"id":"2","type":"terms","schema":"segment","params":{"field":"host.raw","size":1000,"order":"desc","orderBy":"1"}}],"listeners":{}}",
"description": "",
"title": "Events per host",
"uiStateJSON": "{}",
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{"index":"logstash-","query":{"query_string":{"query":"","analyze_wildcard":true}},"filter":[]}"
}
}

{
"visState": "{"title":"New Visualization","type":"histogram","params":{"shareYAxis":true,"addTooltip":true,"addLegend":true,"scale":"linear","mode":"stacked","times":[],"addTimeMarker":false,"defaultYExtents":false,"setYExtents":false,"yAxis":{}},"aggs":[{"id":"1","type":"count","schema":"metric","params":{}},{"id":"2","type":"date_histogram","schema":"segment","params":{"field":"@timestamp","interval":"auto","customInterval":"2h","min_doc_count":1,"extended_bounds":{}}},{"id":"3","type":"terms","schema":"group","params":{"field":"host.raw","size":1000,"order":"desc","orderBy":"1"}}],"listeners":{}}",
"description": "",
"title": "events per host bar",
"uiStateJSON": "{}",
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{"index":"logstash-","query":{"query_string":{"query":"","analyze_wildcard":true}},"filter":[]}"
}
}

So far no clue as to what causes this behavior. Any input is more than appreciated.

[tylersmalley]

Hi Yuval,

Another user posted information today on they were able to successfully create visualizations outside Kibana.

How to put a visualization directly in Kibana/ElasticSearch?

Hello everyone, I tried something and I think it's worth sharing I'm trying to write a script to automatically create a new dashboard according to a specific user (for that, it is the search that changes). More precisely, as a dashboard is made of visualizations, we need to create the visualizations in the first place. I was blocked at this step, creating the visualization directly via a CURL -XPOST request to ElasticSearch. Because in order to do the final dashboard, we need …

Does this help?

[adary (Yuval Adar)]

Not really. My big issue is first creating a default index, and then defining visualisations and dashboards. If i create the default index manually, and after that inject visualisations and dashboards everything is ok. If i try to programatically create the default index and after that inject the data without visiting the gui the problems start.

[adary (Yuval Adar)]

And I seem to have found the solution.

Apparently I have to create the index, import all my dashboards and visualisations in a very specific order of actions:

1. perform a "search"
2. import the visualisations
3. import the dashboard
4. create the index (and it has to be the proper index with all fields)
5. set default index

Best way to do all this is to create everything manually, export all configurations to json (I used the dump script from https://github.com/elastic/beats-dashboards )

Performing a "search" is a critical component in the whole flow. The way I do it is:
/usr/bin/curl -XPUT http://localhost:9200/.kibana/_mapping/search -d '{"search": {"properties": {"hits": {"type": "integer"}, "version": {"type": "integer"}}}}'

If these steps are done in any other order, kibana is not usable.

This was tested on 4.4.1 4.5.0 and 5.0.0-alpha setups

Hope it helps someone