Jupyter -> Vega -> Kibana Visualization Error

[vikram_singh]

Hi,
I want to visualize data from jupyter to Vega to Kibana.
I got a reference How to bring Jupyter Notebook visualizations to Kibana dashboards for data science | Elastic Blog

and GitHub - walterra/jupyter2kibana: A Workflow for Data Scientists to bring Jupyter Notebook Visualizations to Kibana Dashboards

But it is not working.

So finally, I found this

Jupyter -> Vega -> Kibana visualization (without disabling security). How? - Elastic Stack / Kibana - Discuss the Elastic Stack

But I did not understand in KIBANA visualization api.
I am implementing it into mylocal computer without elasticsearch version 8.4.1 and I disabled login security in elasticsearch.yml.
When I used below code into jupyter to send visualization to kibana.

    import eland as ed
    import datetime
    import altair as alt
    import eland as ed
    import json
    import numpy as np
    import matplotlib.pyplot as plt
    import vega_datasets
    from elasticsearch import Elasticsearch
    
    #cloud_id = "secret"
    #http_auth = ("username", "password")
    #es = Elasticsearch(cloud_id=cloud_id, http_auth=http_auth)
    es=Elasticsearch(["http://localhost:9200"])
    data = vega_datasets.data
    pd_df = data.cars()
    
    
    url = 'http://localhost:9200/pandas_to_eland/_search?size=1000'
    #print(url)
    url_data = alt.Data(url=url, format=alt.DataFormat(property='hits.hits',type='json'))

    fields = ed_df.columns

    rename_dict = dict((a, 'datum._source.'+a) for a in fields)

    chart = alt.Chart(url_data).transform_calculate(**rename_dict).mark_circle(size=8).encode(
        alt.X(alt.repeat("column"), type='quantitative'),
        alt.Y(alt.repeat("row"), type='quantitative'),
        color='Origin:N'
    ).properties(
        width=150,
        height=150
    ).repeat(
        row=['Horsepower',],
        column=['Miles_per_Gallon']
    ).interactive()

    chart

    
    def saveVegaVis(client, index, visName, altairChart, resultSize=100, timeField=True):
        chart_json = json.loads(altairChart.to_json())
        visState = {
          "type": "vega",
          "aggs": [],
          "params": {
            "spec": json.dumps(chart_json, sort_keys=True, indent=4, separators=(',', ': ')),
          },
          "title": visName
        }
        visSavedObject={
            #"visualization" : {
              "title" : visName,
              "visState" : json.dumps(visState, sort_keys=True, indent=4, separators=(',', ': ')),
              "uiStateJSON" : "{}",
              "description" : "",
              "version" : 1,
              "kibanaSavedObjectMeta" : {
                "searchSourceJSON" : json.dumps({
                  "query": {
                    "language": "kuery",
                    "query": ""
                  },
                  "filter": []
                }),
              }
            #},
            #"type" : "visualization",
            #"references" : [ ],
            #"migrationVersion" : {
            #  "visualization" : "8.0.0"
            #},
            #"updated_at" : datetime.datetime.now().strftime("%Y-%m-%dT%H:%M:%S.000Z")
        }
        import requests
        myurl = 'http://localhost:5601/api/saved_objects/visualization'
        headers = {
            'kbn-xsrf' : 'true',
            'Content-Type' : 'application/json'
        }
        x = requests.post(myurl, data=json.dumps(visSavedObject), headers=headers)
        print(x.text)
    
    
        #return client.index(index=index,id='visualization:'+visName,body=visSavedObject)
    
    saveVegaVis(es, 'test_visuals', 'def-vega-cars-1', chart, resultSize=1000)

It gives me an error

{"statusCode":400,"error":"Bad Request","message":"[request body.attributes]: expected value of type [object] but got [undefined]"}

I did not understand what is happening.
Kindly help me on this issue.

Thanks in advance,
Vikram Singh

[vikram_singh]

Hi Elastic Community,
Has anyone implemented this type visualization.

[vikram_singh]

Hi,
Any solution!

[walterra (Walter Rafelsberger)]

Thanks for holding out! The referenced Jupyter Notebooks were a bit outdated unfortunately.

They were writing to the .kibana index directly which is no longer recommended/supported.

I updated the first notebook to make use of the Kibana API with support for authentication and using the endpoint to create a visualization: jupyter2kibana/viz-1a-flights-histogram.ipynb at master · walterra/jupyter2kibana · GitHub

I'm reposting the Jupyter Notebook cells here that create the Kibana visualization, note they do not work stand-alone but need to be run as part of the whole notebook.

Helper function to save the Vega visualization:

def saveVegaVis(index, visName, altairChart):
    chart_json = json.loads(altairChart.to_json())
    chart_json['data']['url'] = {
        "%context%": True,
        "index": index,
        "body": { 
            "size": 10000
        }
    }

    visState = {
      "type": "vega",
      "aggs": [],
      "params": {
        "spec": json.dumps(chart_json, sort_keys=True, indent=4, separators=(',', ': ')),
      },
      "title": visName
    }

    visSavedObject={
        "attributes" : {
          "title" : visName,
          "visState" : json.dumps(visState, sort_keys=True, indent=4, separators=(',', ': ')),
          "uiStateJSON" : "{}",
          "description" : "",
          "version" : 1,
          "kibanaSavedObjectMeta" : {
            "searchSourceJSON" : json.dumps({
              "query": {
                "language": "kuery",
                "query": ""
              },
              "filter": []
            }),
          }
        },
    }
    
    return requests.post(
        es_config['kibana_client'] + '/api/saved_objects/visualization/' + visName,
        json=visSavedObject,
        auth=(es_config['user'], es_config['password']),
        headers={"kbn-xsrf":"jupyter2kibana"},
        # Only in development environments with self signed certificates fall back to use `verify=False`
        verify=False
    )

Command example to save the Vega visualization:

saveVegaVis('kibana_sample_data_flights', 'def-vega-1', url_charts)

Hope this helps adapting your own code!

[vikram_singh]

Thanks for help.
It solved my problem.