Submitting jobs to apache spark

Elastic Stack Kibana
1

Hi!

I am now creating a custom kibana plugin.

** What I want to do **
I want to submit jobs to Apache spark from using result that were queried from elasticsearch.
By submitting the job , I will generate predictive analytic data points using spark library .

I have read below article and creating a new server API looks to be a starting point.

Is there a way to call spark-submit from the Kibana server?
Or is there any api's in Spark which I can call from Kibana server?

Thanks,
Yu Watanabe

2

I'm not too sure about what exists on the Apache Spark side, but you should be able to create your own server APIs which invoke the Spark APIs.

If I'm understanding correctly, you'd like to submit jobs to Spark every time there is a query to Elasticsearch?

3

Hello @lukas

Yes . Your understanding is correct. I am planning to connect to spark whenever there is a
search executed from Kibana interface and accessed to elasticsearch.