Automatically visualize data on kibana from input?

Elastic Stack Kibana
1

I'm wondering if there's a way to automatically visualize data on KIBANA. ? My logstash will receive frequent new inputs, and i need to know if there's way to automatically create dashboards ? may be a way to define in templates ?

2

There is no fully automated way to do so - however you could build something like this using a script and the saved object api of Kibana: Saved objects APIs | Kibana Guide [7.12] | Elastic

A bit like this:

It's not an out-of-the-box solution, but you would have a lot of control this way.

3

Thank you @flash1293 , i will go through these documents and get back incase of questions :slight_smile:

Cheers!

4

i was able to view the fields in my index via GET method . Not sure if this is correct , please correct me if this is wrong

GET log_sample/_field_caps?fields=*

{
  "indices" : [
    "log_sample"
  ],
  "fields" : {
    "CustID" : {
      "text" : {
        "type" : "text",
        "searchable" : true,
        "aggregatable" : false
      }
    },
    "Forward_hostname" : {
      "text" : {
        "type" : "text",
        "searchable" : true,
        "aggregatable" : false
      }
    },
    "Req_HTTP_Status" : {
      "text" : {
        "type" : "text",
        "searchable" : true,
        "aggregatable" : false
      }
    },
    "Req_Transfer_time.keyword" : {
      "keyword" : {
        "type" : "keyword",
        "searchable" : true,
        "aggregatable" : true
      }
    },
    "RequestID" : {
      "text" : {
        "type" : "text",
        "searchable" : true,
        "aggregatable" : false

How would one be able to turn these fields into visualization / dashboard saved objects ?

5

It depends on how exactly you want them visualized - as you know Kibana offers a lot of ways to do so.

Check out how your existing Visualizations are structured (by exporting them). In most cases you probably want to replace the field name mentioned in the export by the new field names.

6

Thank you @flash1293 , in my case , the field names would remain the same . Only the data would keep changing , as new log files are taken in as input.

I will try exporting them and get back in case of doubts . Thanks

7

Wait, your fields will stay the same? Maybe I misunderstood your question then.

My logstash will receive frequent new inputs, and i need to know if there's way to automatically create dashboards

What do you mean by that? If there's just new data coming in, you can simply change the time range in the top right to visualize your new data in your existing dashboards. Dashboards and visualuzations are just a piece of configuration, the data is always loaded from Elasticsearch directly. It's not saving the actual data points, just a description how to fetch the data and how to visualize it.

8

Yes, the fields will stay the same . But the Data that will be displayed will change .

The idea is to treat each input file as different when a user places a log file,but display data in the same kind of visualizations

9

Then it's way easier than that. Index everything into the same index, then put the source of the data into each document (it's common to have a dataset field), then simply add a filter like dataset: thatFile or dataset:thisFile to your dashboard.

This allows you to use the same dashboard and switch between different data sources.

Related: Logstash: how to add file name as a field? - Stack Overflow

10

Thank you , i will try this .

11

okay , so following the stack overflow methods, i was able to include the file being processed in the filter section. Do we add filter "dataset:thisFile " on the discover tab of kibana ?

{
  
              "filename" => "request_98485.20.1619172488.1619176238.gz",
         "Req_Client_IP" => "2600:1700:d91:3500:9cff:d071:cafb:ecec",
     "Req_Transfer_time" => "0",
         "Req_epoch_time" => "1619176208.076",
       "Req_HTTP_Status" => "202",
          

![Screen Shot 2021-05-11 at 5.09.05 PM|690x119](upload://hzfBz6SHnflYNo9o9sc4Y986mft.jpeg)
12

You can add it in Discover, Visualize and Dashboard.

It's the little "Add filter" button below the search bar:
Screenshot 2021-05-11 at 14.41.47

13

Okay, so this would enable us to use the same dashboard for different data . Can we create newer dashboards based on existing ones for each user automatically ?

14

The users could just use the single dashboard and set their filter there. As the filter is part of the URL, you can maintain a list of links to the same dashboards with multiple filters pre-set. But there's no way to do this automatically.

You can also use the "controls" visualization to create a form element in a panel to make it easier for users to pick a filename: Enhance dashboards | Kibana Guide [7.12] | Elastic

15

Our Idea is to have users upload a log file based on timeframes to a location where logstash will process it , however we do not want them to use "search" filters. But need them to only upload the log file and file would be processed and dashboard created automatically .

We have the filters written and pipelines configured , but i couldn't find a way to autocreate dashboards based on the ones we create the first time as a reference or a baseline.

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.