Create Kibana visulaization on the fly using python script

[Bhairavi]

I have a requirement of creating kibana visualization when a user writes a query like "apache logs for March". I want it to happen dynamically using Python script.
I've tried submitting Kibana JSON request through Elasticsearch's .kibana index and it worked if "query" value is static.

 import config
from elasticsearch import Elasticsearch,RequestsHttpConnection
es=Elasticsearch(hosts=[{'host':config.hostName,'port':config.port}],http_auth=
(config.userName,config.password),connection_class=RequestsHttpConnection)

feeds = es.index(index=".kibana",doc_type="visualization", body={
      "title": "test kibana1",
      "visState": """{"aggs":[{"enabled":true,"id":"1","params":
{},"schema":"metric","type":"count"},{"enabled":true,"id":"2","params"
{"customInterval":"2h","extended_bounds"
{},"field":"@timestamp","interval":"d","min_doc_count":1},"schema":"segment","type":"d
ate_histogram"}],"listeners":{},"params"
{"addLegend":true,"addTooltip":true,"isDonut":false,"legendPosition":"right"},
"title":"test
kibana1","type":"pie"}""",
"uiStateJSON": "{}",
"description": "",
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": """{"index":"logstash-yyyy.mm.dd","query":{"match_all":
{}},"filter":[{"$state":{"store":"appState"},"meta":
{"alias":null,"disabled":false,"index":"logstash-
yyyy.mm.dd","key":"message","negate":false,"type":"phrase","value":”apache logs for
march”},"query":{"match":{"message":{"query":”apache logs for march”}}}}]}"""
}
})
	
print(feeds)		

After executing this I am able to see the visualization in Kibana.

But I am aiming to pass string user input to this "query" parameter for which I am getting visualization created message but it's just creating a name for the visualization on Kibana side because it is not accepting the variable which I am trying to pass.

 import config
from elasticsearch import Elasticsearch,RequestsHttpConnection

es=Elasticsearch(hosts=[{'host':config.hostName,'port':config.port}],http_auth=
(config.userName,config.password),connection_class=RequestsHttpConnection)

post = input("Enter post:")

feeds = es.index(index=".kibana",doc_type="visualization", body= {

      "title": "test kibana1",
      "visState": """{"aggs":[{"enabled":true,"id":"1","params":
{},"schema":"metric","type":"count"},{"enabled":true,"id":"2","params":
{"customInterval":"2h","extended_bounds":
{},"field":"@timestamp","interval":"d","min_doc_count":1},"schema":"segment",
"type":"date_histogram"}],"listeners":{},"params":
{"addLegend":true,"addTooltip":true,"isDonut":false,"legendPosition":"right"},
"title":"test kibana1","type":"pie"}""",
      "uiStateJSON": "{}",
      "description": "",
      "version": 1,
      "kibanaSavedObjectMeta": {
        "searchSourceJSON": """{"index":"logstash-yyyy.mm.dd","query":{"match_all":
{}},"filter":[{"$state":{"store":"appState"},"meta":
{"alias":null,"disabled":false,"index":"logstash-
yyyy.mm.dd","key":"message","negate":false,"type":"phrase","value":post},"query":
{"match":{"message":{"query":post}}}}]}"""
      }
    })

Is there any way to achieve this? Please help.

[timroes]

"because it is not accepting the variable which I am trying to pass." could you please elaborate what you mean by not accepting the variable? Will the object be stored in the .kibana index? If you go in Kibana to management -> Saved Objects, do you find it there? Could you post how the saved object looks there? At what part does it fail exactly?

[Bhairavi]

"because it is not accepting the variable which I am trying to pass." - by this I mean if the variable "post" reads the user input say "apache logs for March" and I am passing it in SearchSourceJSON's query it should execute the query based on the input, a user has given to the "post" variable but it is not working in that way.
Yes, the object will be stored in the .kibana index and I can find it inside the saved objects even if it is failing. The error occurs at SearchSourceJSON part.

[timroes]

This is rather a Python question then a Kibana question in your case you just write the string post in your JSON instead of the content of the variable.

You should have a look into Python String format: https://pyformat.info/

Your python should in the end look something like:

"visState": """ ... "query": "{}" ... """.format(post)

[Bhairavi]

I've tried formatting the string as you said but it rather gives me KeyError: '"index"' error.

But is there any other approach with Kibana which can help this use case, where based on user's string query, dynamically Kibana visualizations are being created using script?

[timroes]

The failing Python is still only python, so nothing Kibana/ES specific in there.

In Kibana itself you can't "dynamically create and save" visualizations based on a query. But of course you can just create that visualization without the query, and then place it on any dashboard you want and as soon as the user puts a search in the query bar of the dashboard, the visualization will only visualize these documents.

I don't think i've understood your use-case well enough so far, to provide a better answer to that.

[Bhairavi]

Use case: We are working on a SAAS based application where we are trying to create a complete enterprise search solution on top of Elasticsearch. And we are intending to show the data in form of dashboards on our front-end with the help of Kibana.
When a user will perform the search query, our goal is to take that query and show respective visualizations created by Kibana along with the results showed in this picture.

Is there any API or method which Kibana supports to do so. Please let me know. Any help would be appreciated. Thank You

[timroes]

To be honest, if you are building that custom search solution and want to integrate charts in that, I would rather recommend not using Kibana for that. Since you are anyway developing some web frontends, I would recommend just loading the data from Elasticsearch via its API (the same as Kibana just using aggregations to load the data) and using any of the huge amount of charting libaries out there (chart.js, react-vis, ...) to visualize the data.

Embedding Kibana like you are trying into a different web solution will usually only will cause trouble in the long run and for sure isn't the most performent solution, since we need to load all of Kibana just to show a chart in your solution.