Making API calls from Kibana in realtime

Elastic Stack Kibana
1

Hello,

Hope everyone is doing great!

This is Sri, i just started exploring ELK stack. I've following requirement, hope i can get inputs from the ELK experts on this.

I'm working on the real time dashboard in Kibana. I've processes running on different tools/technologies [One of the example is jobs running in ETL] and the jobs are running 24/7. One of the requirement for my dashboard is to show the processes currently running on these different tools in real time. We have API's available from every tool to get the list of current running jobs. Now i need to call these API's to show the current running jobs in Kibana dashboard whenever user accessing the dashboard.

Would it be possible to make call to external API's from Kibana in real time while accessing dashboard and show the response of these API's in the dashboard? If yes, can you please provide details how to implement this?

Appreciate your help!

Thanks,
Sri

2

What do you mean by real-time?
1 millisecond, 1 second, few seconds?

Kibana connects to Elasticsearch in order to show & generate graphs.
The data is stored in Elasticsearch

You could:

  • use Logstash to poll your external API via Http and send the result in Elasticsearch
  • use Heartbeat to poll your external API via Http and send the result in Elasticsearch
  • use any programming language to poll your external API via Http and send the result in Elasticsearch using any of our official clients or use the raw Elasticsearch HTTP APIs
  • If you have a Gold or Platinum license you might use Watcher to setup a periodic job to poll your external API via Http and store the result in Elasticsearch
3

Thank you @Luca_Belluccini for your prompt reply!

What I meant by "real-time" is when the user accessing the dashboard, need to show the jobs that are currently running at that time [user can access the dashboard through out the day at any time]. In other words, I wanted to give the real time view of current running jobs to my users.

So i need to setup a job to poll the external API's for every few seconds to get the running jobs information and send these results to Elasticsearch and take the latest result set and show it in the dashboard? If i poll API's for every few seconds [for example 20 -30 seconds], wouldn't it be too many documents written to Elasticsearch and will i run into any performance issues?

Instead of polling, would it be possible to call external API at the time of accessing the dashboard from Kibana and send the API response results to Elasticsearch and show these results in Kibana? The reason why i wanted to go this route [if its possible], it will reduce the number of external API calls and the load on my external API server [for example ETL server] will be less otherwise constantly for every few seconds [if i poll for every 20 seconds, total 4320 calls per day to my ETL server to get the details] i'll be making call to external API to get the data.

Yes we are using Watcher.

4

Elasticsearch is designed to handle a lot of data.

I would suggest using Heartbeat for polling your APIs.
Some additional documentation is available here.
This blog post can help.

Another approach is using Logstash as explained in this blog post.

With Watcher you can use the HTTP input (as explained here) and index the data using the indexing action.
But still, Watcher runs all the time, it is running even if Kibana is accessed by anyone.

To conclude, Kibana doesn't work like that and always remember the data is stored in Elasticsearch.

If your ETL "Status" API supports Prometheus, you might think pulling data from Prometheus as explained here.

5

Thank you for the valuable inputs @Luca_Belluccini.

I've the enough information to start implementing one of the option. I've one final question.

If i go with Heartbeat approach then it need to be installed and API need to be configured in my remote server [in this case ETL server] and ship the API response to ELK. By looking at the documentation, Heartbeat is mainly used to monitor the services running. Can i configure parameters while configuring HTTP using heartbeat?

Since the Heartbeat need to be installed on remote server, do i need to consider anything for load balancing on the remote server because the API call will be executed for every few seconds.

If i go with Logstash approach then the HTTP poller need to be setup as my Logstash input from ELK cluster which will make a call to my external API and the response will be stored as logstash events. Am i correct?

Thanks!

1 Like
6

Both Heartbeat and Logstash can be started on any server which can reach out the ETL Server API via HTTP and which can reach the Elasticsearch cluster to write the responses.

Both Heartbeat and Logstash get the response and send a document to Elasticsearch.

By parameters you mean the payload you want to send to the ETL API?
Yes, you can customize the request (see doc).
Example to send POST http://api-etl-server:8080/status with a JSON payload { "oneparam": 123 } with HTTP authentication every 5 minutes.

Heartbeat:

- type: http
  schedule: '@every 5m'
  urls: ["http://api-etl-server:8080/status"]
  username: myuser
  password: mypassword
  check.request:
    method: POST
    headers:
      'Content-Type': 'application/json'
    body: '{ "oneparam": 123 }'

Logstash:

input {
  http_poller {
    urls => {
      status_etl=> {
        method => post
        user => "myuser"
        password => "mypassword"
        url => "http://api-etl-server:8080/status"
        body => '{ "oneparam": 123 }'
        headers => {
          "Accept" => "application/json"
          "Content-Type" => "application/json"
        }
     }
    }
    request_timeout => 60
    schedule => { "every" => "5m" }
    codec => "json"
  }
}
7

Yes API payload.

Thank you for the pseudo code snippet. You have provided lot of information to get started on this. Once again thank you very much!

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.