Is there any current thoughts on taking output from a kibana query, and being able to pipe it or take it to an ipython notebook? I am highly interested in this feature, or potentially adding a feature to allow the output of a query be piped to a python function/library for quick stats. primarily, i am itnerested in taking the output of the search and piping it to a pandas dataframe for quick statistics on certain fields.
Python has an elasticsearch library that can be used in jupyter or ipython notebooks. Does that not help in your use case?
It's an interesting idea, it'd should be possible as a plugin.
Im more so trying to give a functionality to the search feature of kibana, similar to splunks ability to pass data to a stats function like below:
splunk: search mystuff | stats count by field1, field 2
thoughts for kibana: search mystuff | (pandas df).countby('field1, field2')
It looks as though splunk does something similar because they take the output of their search and pass it to another page (statistics tab), im assuming is just a version of output from a function.
If we could do something similar, we could be able to pass the data to python, then we would have very similar abilities to conduct basic stats, evaluations of new fields, splitting fields, regex support and what not. pandas supports 99% of the use cases i have seen in this capability and i think passing the search output from elk to a pandas dataframe instead of offering some app to do so, would make it inherently faster and more efficient for a majority of the users.
That is an interesting take. I believe an app similar to Timelion can be developed specifically for Data Science support in Kibana. This would be an interesting way.
I also use Kibana simply for dash-boarding and I use pandas, numpy, and scipy to do all the analysis, and send the data back to another index in elasticsearch. I think a major area where the ELK stack lacks is the analysis component. I hope Canvas will solve some of the problems in that area.
1 Like
I also believe that this would be helpful. Kibana, also gives the data as a dictionary, pandas can easily create a dataframe from the dictionary.
Maybe open an enhancement request on Github?
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.