How to read elasticsearch data into custom visualisation?

[reillye (Eoin)]

I am trying to build a custom visualisation in Kibana. I have used to plugin generator to generate a base plugin, but I have struggled to find up to date examples on how to create a custom visualisation using it. I have tried following the examples from logz.io, and [https://www.timroes.de/writing-kibana-plugins-custom-applications](Tim Roes' blog ) but they are over 3 years old now and haven't been of much help.

The closest example I was able to find was this elastic blog which is about a year and a half old, and they even provided full code on their github here. But even after copying their code line for line, I was unable to get their visualisation to display via the plugin.

The visualisation I am trying to create is known as a Skyplot, which plots satellite positions around the earth at points in time. Below is a screenshot of what I have been able to generate so far, using the react-plotly.js library to create the visualisation:

My issue is that this is currently just using static data I've hard coded in. I'm looking for examples/help on how to integrate Elasticsearch data from an index into this. I want to be able to choose a certain time period, query for this data in elastic, and then populate it on the visualisation.

I realize I've asked for a lot of information here, but I'm struggling to find examples on how to implement this, so any advice is appreciated!

I can provide the code I've written if necessary, although the only file I have changed is the components/main.js where I have added the chart generated code and the static data.

[Bill_McConaghy]

You will want to set up an API endpoint on Kibana that your plugin can hit passing the date as a parameter. This is an example of how to set up the API endpoint in code: https://github.com/elastic/kibana/blob/master/src/legacy/server/sample_data/routes/list.js

The callWithRequest in there wraps the Elasticsearch JS API, which is documented here: https://github.com/elastic/elasticsearch-js

So your plugin client side code will call your endpoint, the endpoint will call ES with the query you want to run and return the data, and then your plugin will render that data.

Hope this is helpful.

[reillye (Eoin)]

Thanks for your reply. I have been trying to implement what you have suggested, and I have a much better understanding of how this works now, but I'm still a bit stuck.

I have set up the following route, to simply return all the indices to the client:

export default function (server) {

  // We can use this method, since we have set the require in the index.js to
  // elasticsearch. So we can access the elasticsearch plugins safely here.
  const call = server.plugins.elasticsearch.callWithRequest;

  server.route({
    path: '/api/skyplot_plugin_2/example',
    method: 'GET',
    handler(req, reply) {
      call(req, 'cluster.state').then(function (response) {
        // Return just the names of all indices to the client.
        reply(
          { indices: (Object.keys(response.metadata.indices)) }
        );
      });
    }
  });
}

But it is giving me the following error:

Debug: internal, implementation, error
    TypeError: call is not a function
    at handler (C:/kibana-extra/skyplot_plugin_2/server/routes/example.js:22:7)
    at Object.internals.handler (C:\kibana\node_modules\hapi\lib\handler.js:96:36)
    at request._protect.run (C:\kibana\node_modules\hapi\lib\handler.js:30:23)
    at module.exports.internals.Protect.internals.Protect.run (C:\kibana\node_modules\hapi\lib\protect.js:64:5)
    at exports.execute (C:\kibana\node_modules\hapi\lib\handler.js:24:22)
    at each (C:\kibana\node_modules\hapi\lib\request.js:384:16)
    at iterate (C:\kibana\node_modules\items\lib\index.js:36:13)
    at done (C:\kibana\node_modules\items\lib\index.js:28:25)
    at Hoek.once (C:\kibana\node_modules\hapi\lib\protect.js:52:16)
    at wrapped (C:\kibana\node_modules\hoek\lib\index.js:879:20)
    at done (C:\kibana\node_modules\items\lib\index.js:31:25)
    at Function.wrapped [as _next] (C:\kibana\node_modules\hoek\lib\index.js:879:20)
    at Function.internals.continue (C:\kibana\node_modules\hapi\lib\reply.js:108:10)
    at C:/kibana/src/server/http/xsrf.js:45:12
    at Items.serial (C:\kibana\node_modules\hapi\lib\request.js:403:22)
    at iterate (C:\kibana\node_modules\items\lib\index.js:36:13)
    at done (C:\kibana\node_modules\items\lib\index.js:28:25)
    at Function.wrapped [as _next] (C:\kibana\node_modules\hoek\lib\index.js:879:20)
    at Function.internals.continue (C:\kibana\node_modules\hapi\lib\reply.js:108:10)
    at C:/kibana/src/server/http/version_check.js:36:12
    at Items.serial (C:\kibana\node_modules\hapi\lib\request.js:403:22)
    at iterate (C:\kibana\node_modules\items\lib\index.js:36:13)

I'm basing this off examples from here and here

I have also tried the example from your suggestion using const { callWithRequest } = request.server.plugins.elasticsearch.getCluster('data');, replacing data with my cluster name elasticsearch, and that results in the whole server crashing.

Can you or anyone help me regarding this issue?

[reillye (Eoin)]

I resolved this by using const { callWithRequest } = request.server.plugins.elasticsearch.getCluster('data'); . I replaced the data part with my own cluster, but this was incorrect, and you need to use data.

I can now retrieve metadata from the cluster, such as the index names etc., but I'm not sure how you actually search for data. Can anyone help with this? Is there a working example I can base it on?