New to Kibana, a question on extensability using R and Python libaries

[Jonathan_Dunne]

Hello,

I'm looking at Kibana and Vega to potentially using it for use within my PhD as I am analysing large volumes of data. For the most part, both applications appear useful. However, I noticed from the video demo's the focus is getting data and visualising within a dashboard.

However, as a data scientist, I want to do more than plot the data on a dashboard. For example, I may need to run regression analysis (I'd like to see GoF charts) I may require running some time series prediction using ARIMA (could l make calls to an R library and surface the values within a Kibana Time Series) etc. I'm wondering if it's possible to harness the power of robust R and Python libraries and surface these outputs artefacts within kibana

In other words, is there a way to wrap either R or python Code inside Vega and surface within Kibana? Or would I need to write a specific library for each particular piece of piece of statistical analysis I require?

Regards
Jonathan

[thomasneirynck]

hi @Jonathan_Dunne,

that's a lot of questions.

Short answer, no, you cannot run R or python code inside Vega. In essence, it's a markup language and defined pretty strictly, without bindings to other languages. It does have a python implementation https://altair-viz.github.io/.

What you can consider is doing all your data analytics with the tools you're familiar with, e.g. R and Python, and then index results of those analyses into Elasticsearch (e.g. one new index per new result). Then you can just generate your graphs from that data inside Kibana (e.g. using Vega).

[Jonathan_Dunne]

Hi Thomas,

Thanks for the response. I understand your point about doing specific satistical analysis natively in R / Python, then index resutls in Elasticsearch.

If I use the example of doing linear regression of 2 or more varibles. If I index the results to Elasticsearch, would it be possible to graph my linear model resutls in Vega using the 4 standard Linear regression Goodness of fit graphs (i.e. Residuals Vs Fitted, Normal Q-Q, Scale-Location and Residuals Vs Leverage)

At a high level these 4 graphs are essentially scatterplots, however they all contain a number of additional overlays (i.e. normal line in the case of a Q-Q plot and Cook's distance curves for the residuals Vs Leverage plot).

[thomasneirynck]

hi @Jonathan_Dunne,

in Kibana, you would be able to do the scatter plots (e.g. with the Vega-visualization), but would not be able to do the overlays (e.g. linear regression line etc...).

For this, I think you might be hitting the limitations of Kibana. it's not quite a data-analytics platform and scientific visualizations are not its strength.

You can also submit feature request for these kind of things here: https://github.com/elastic/kibana/issues/new

Nonetheless, I would take a look at the Vega-visualization, as you might be able to do the kind of charts in there that you may not be able to do with a platform like R.