Using Kibana REST API to generate visualization

Hello everyone,

I'm looking to generate dashboards and visualizations using the Kibana API and run a Python script.
I watch a lot of other threads but couldn't find what I needed.

The context:

  • A python script parses many CSV files to merge the data, computes new columns then pushes the data to the elastic API (it works)
  • Another Python script uses a YAML configuration file to generate dashboards with visualization.

I want to override this manual import to avoid errors and be able to automatically update my dashboards when the YAML file is updated.

Do you know and do you have an example of visualization creation in Kibana with API?

Thanks !


Simple curl command works well.

curl --user USERNAME:PASSWORD -k -X GET https://MYURL/s/MYSPACE/api/saved_objects/_find?type=index-pattern&search_fields=title&search=DevSecOps

This is an extract of my current script:

    def post_panel(self, data, id):
        cmd = f"curl --user {self.__username_write}:{self.__password_write} -k -X -H 'Content-Type: application/json' -H 'kbn-xsrf: true' POST {self.kibana_url}/s/{self.space}/api/saved_objects/visualization/{id}  -d '{data.to_json()}'"
        print(cmd)
        stream = os.popen(cmd)
        r = stream.read()
        r = json.loads(r)
        print(r)

Terminal print

INFO: Python Data Injector - U2P - 20/07/21
INFO: Starting...
INFO: Logging level set to: verbose
INFO: Namespace(env='Dev', c='./resources/config.yaml', dashboard=False, purge=False, faker=False, verbose=True, validate=False)
C:\Users\U2P\PycharmProjects\pythonProject\venv\lib\site-packages\elasticsearch\connection\http_urllib3.py:209: UserWarning: Connecting to https://MYURL:443 using SSL with verify_certs=False is insecure.
  warnings.warn(
POST
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   679  100   679    0     0    679      0  0:00:01 --:--:--  0:00:01  1550
'search_fields' n'est pas reconnu en tant que commande interne
ou externe, un programme ex�cutable ou un fichier de commandes.
'search' n'est pas reconnu en tant que commande interne
ou externe, un programme ex�cutable ou un fichier de commandes.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   679  100   679    0     0    679      0  0:00:01 --:--:--  0:00:01  3947
'search_fields' n'est pas reconnu en tant que commande interne
ou externe, un programme ex�cutable ou un fichier de commandes.
'search' n'est pas reconnu en tant que commande interne
ou externe, un programme ex�cutable ou un fichier de commandes.
curl --user USERNAME:PASSWORD -k -X -H 'Content-Type: application/json' -H 'kbn-xsrf: true' POST https://MYURL/s/MYSPACE/api/saved_objects/visualization/90671daf-edfc-5298-41b0-0dfea6130fbd  -d '{
  "attributes": {
    "description": "Graph Business importance",
    "kibanaSavedObjectMeta": {
      "searchSourceJSON": {
        "query": {
          "query": "",
          "language": "kuery"
        },
        "filter": [
          {
            "meta": {
              "negate": true,
              "type": "phrase",
              "key": "application.Business Importance Label",
              "params": {
                "query": ""
              },
              "disabled": false,
              "alias": null,
              "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index"
            },
            "query": {
              "match_phrase": {}
            },
            "$state": {
              "store": "appState"
            }
          }
        ],
        "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.index"
      }
    },
    "title": "[Test] Pie chart",
    "uiStateJSON": {
      "vis": {
        "colors": {
          "Critical": "#4287f5",
          "Standard": "#565d69",
          "High critical": "#b02eac",
          "Low": "#ff6242",
          "Not applicable": "#42ff4c"
        }
      }
    },
    "version": 1,
    "visState": {
      "title": "[Test] Pie chart",
      "type": "pie",
      "aggs": [
        {
          "id": "1",
          "enabled": true,
          "type": "count",
          "params": {},
          "schema": "metric"
        },
        {
          "id": "2",
          "enabled": true,
          "type": "terms",
          "params": {
            "field": "application.Business Importance Label",
            "orderBy": "_key",
            "order": "asc",
            "size": 5,
            "otherBucket": false,
            "otherBucketLabel": "Other",
            "missingBucket": false,
            "missingBucketLabel": "Missing",
            "customLabel": ""
          },
          "schema": "segment"
        }
      ],
      "params": {
        "type": "pie",
        "addTooltip": true,
        "addLegend": true,
        "legendPosition": "top",
        "isDonut": false,
        "labels": {
          "show": false,
          "values": true,
          "last_level": true,
          "truncate": 100
        }
      }
    }
  },
  "id": "90671daf-edfc-5298-41b0-0dfea6130fbd",
  "migrationVersion": {
    "visualization": "7.11.0"
  },
  "references": [
    {
      "id": "c1684c20-f5d1-11eb-bd95-ed7eb02bc1bb",
      "name": "kibanaSavedObjectMeta.searchSourceJSON.index",
      "type": "index-pattern"
    },
    {
      "id": "c1684c20-f5d1-11eb-bd95-ed7eb02bc1bb",
      "name": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index",
      "type": "index-pattern"
    },
    {
      "id": "dda38f60-f9b6-11eb-bd95-ed7eb02bc1bb",
      "name": "tag-ref-dda38f60-f9b6-11eb-bd95-ed7eb02bc1bb",
      "type": "tag"
    }
  ],
  "type": "visualization",
  "updated_at": "2021-08-16T15:08:22.682926",
  "version": "WzQ3MDcsMV0="
}'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: 'Content-Type
  0     0    0     0    0     0      0      0 --:--:--  0:00:20 --:--:--     0curl: (7) Failed to connect to application port 80: Timed out
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: true'
  0     0    0     0    0     0      0      0 --:--:--  0:00:20 --:--:--     0curl: (7) Failed to connect to POST port 80: Timed out
100     2    0     0  100     2      0      2  0:00:01 --:--:--  0:00:01    12
Traceback (most recent call last):
  File "C:\Users\U2P\PycharmProjects\injector-python\main.py", line 79, in <module>
    es.post_panel(d.dashboards[0].panels[0])
  File "C:\Users\U2P\PycharmProjects\injector-python\elastic.py", line 117, in post_panel
    r = json.loads(r)
  File "C:\Users\U2P\AppData\Local\Programs\Python\Python39\lib\json\__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "C:\Users\U2P\AppData\Local\Programs\Python\Python39\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Users\U2P\AppData\Local\Programs\Python\Python39\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

There is no exposed API for creating dashboards, visualizations, or lens. The stored state of dashboards, visualization, and lens objects are intentionally not published or documented. These details are considered internal and change from minor too minor. You can open an issue in the Kibana repo to expose a API for creating these saved objects.

You could always brute force the issue and analyze the structure of the saved objects you want to create via and API. Just go into it knowing that your solution will be fragile and will need constant updating to work between minors.

An easy way to view the structure of saved objects is to query for them in Console. Or you could look at example saved objects from sample data.

// Query for saved objects
GET .kibana/_search
{
 "query": {
   "match": {
     "type": "dashboard"
   }
 }
}
1 Like

Ok...

Thanks for your answer. I will open a issue about dashboard as code and rework my script.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.