Vega visualization in Kibana - unable to load data json


#1

Hi,

I am a beginner at Vega - I successfully got a "line chart" working with a inline dataset (specified with "Values") and wanted to take it to next step of getting the dataset from a json served via a URL.

When I try that, Kibana visualization area shows me:
Loading failed
http://localhost/jsons/ihs_vega.json

I have enabled
vega.enableExternalUrls: true

My json is being served by nginx on my PC. Both http and https URLs do successfully serve the json when accessed through browser.

In Vega, I tried the url to be http as well as https - both give the same problem. Note: my https server is running with a self-signed certificate (don't know if that needs to be added to some truststore to make it work).

I turned on Kibana verbose logging (logging.verbose: true), but it didn't offer any clue.

I could not find out any other parameters to turn on vega logging/tracing.

Environment Details:
Operating System: Windows 10
Kibana and Elastic version: 6.4

Complete vega code:

{
  "$schema": "https://vega.github.io/schema/vega/v3.json",
  "width": 500,
  "height": 200,
  "padding": 5,
  "autosize": "pad",

  "data": [
    {
      "name": "ihsdata",
      "url": "https://localhost/jsons/ihs_vega.json",
      "format": {
        "type": "json",
        "parse": {"RequestTime": "date"}
      },
    }
  ],

  "scales": [
    { "name": "ixscale", "type": "time",
      "range": "width",
      "domain": {"data": "ihsdata", "field": "RequestTime"} },
    { "name": "iyscale", "type": "linear",
      "range": "height", "nice": true, "zero": false,
      "domain": {"data": "ihsdata", "field": "ResponseTimeMS"} }
  ],

  "axes": [
    {
      "orient": "left",
      "scale": "iyscale",
      "grid": true,
      "_tickCount": 5
    },
    {
      "orient": "bottom",
      "scale": "ixscale",
      "_tickCount": 5
    }
  ],

  "marks": [
    {
      "type": "line",
      "from": { "data": "ihsdata" },
      "encode": {
        "enter": {
          "stroke": {"value": "#F00"}
        },
        "update": {
          "x": {"scale": "ixscale", "field": "RequestTime"},
          "y": {"scale": "iyscale", "field": "ResponseTimeMS"},
          "strokeWidth": {"value": 2},
          "strokeDash": {"value": [15,3]},
          "opacity": {"value": 1},
          "stroke": {"value": "#F00"}
        },
        "hover" : {
          "strokeDash": {"value":[1,0]},
          "stroke": {"value": "#00F"}
        }
      }
    }
  ]
}

ihs_vega.json looks like:

[
  {
    "RequestSize":52,
    "RequestType":"POST",
    "AppServer":"Server4",
    "ResponseTimeMS":29.6,
    "RequestDate":"2018-09-11",
    "RequestTime":"2018-09-11T05:00:00.000+0530",
    "IHS":"IHS3"},
  {
    "RequestSize":51,
    "RequestType":"POST",
    "AppServer":"Server5",
    "ResponseTimeMS":36.0,
    "RequestDate":"2018-09-11",
    "RequestTime":"2018-09-11T05:00:02.000+0530",
    "IHS":"IHS1"},
  {
    "RequestSize":54,
    "RequestType":"GET",
    "AppServer":"Server3",
    "ResponseTimeMS":37.4,
    "RequestDate":"2018-09-11",
    "RequestTime":"2018-09-11T05:00:07.000+0530",
    "IHS":"IHS2"}
]

Can someone point me to the right direction?

Thanks so much.

24/Sep ... adding one more (related) question here
My kibana-vega visualization doesn't work for elasticsearch index as well. It doesn't give any errors, but it doesn't draw the desired picture. When my dataset points to "values", it works properly, but when I use the dataset as

{
    "name": "ihsdata",
    "url": { "index": "ihs-proxy-server*"
           }
}

it doesn't draw the desired picture. I don't know how to troubleshoot this.

I have one index called ihs-proxy-server-2018-09-11 and its structure is as follows:

http://localhost:9200/ihs-proxy-server-2018-09-11/?pretty=true

{
  "ihs-proxy-server-2018-09-11" : {
    "aliases" : { },
    "mappings" : {
      "doc" : {
        "properties" : {
          "@timestamp" : { "type" : "date" },
          "@version" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } },
          "AppServer" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } },
          "IHS" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } },
          "RequestDate" : { "type" : "date" },
          "RequestSize" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } },
          "RequestTime" : { "type" : "date" },
          "RequestType" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } },
          "ResponseTimeMS" : { "type" : "float" },
          "host" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } },
          "message" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } },
          "path" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }
        }
      }
    },
    "settings" : {
      "index" : {
        "creation_date" : "1537265561043",
        "number_of_shards" : "5",
        "number_of_replicas" : "1",
        "uuid" : "0ohmIIhASJqxRgFVRXrvgA",
        "version" : { "created" : "6040099" },
        "provided_name" : "ihs-proxy-server-2018-09-11"
      }
    }
  }
}

Thanks

-- Parag


(kulkarni) #2

cc @nyuriks - any help here ?

Thanks
Rashmi


(Yuri Astrakhan) #3

Hi @deuskars! The best way to troubleshoot something like this is to use browser's built-in debugging tools. There could be a number of things going wrong. Try these things:

  • Open browser developer tools (in Chrome - menu icon, tools, dev tools). Open networking tab. Do some minor vega modification (e.g. add some top-level key like title: graph title and click the "play" button. Observe the networking requests your browser makes, and see if there are any issues - if it gives you the expected result, no http errors, etc.
  • Try uploading your data to some well know well host, e.g. https://gist.github.com/ and use the link to the RAW version of that file. This should work with both http and https.

P.S. this will also help with the Elasticsearch request


#4

Thanks @nyuriks and @rashmi ...

Regarding getting data from json - I was able to move forward a bit - as expected, it was an invalid cert error (since I was using self signed cert). Thanks for the tip about gist.github.com, it was indeed helpful. I am facing some different problems there (cors need to be enabled and facing some problems there).

Regarding getting data from elastic index (which is my real aim) - I am still facing some issues. The tip for F12 debug and then looking at VEGADEBUG.view.data was very useful. I still haven't solved my problem, but at least got a better view into it. The data returned by elasticsearch is much more nested json that my plain json. I tried to add "format": { "property": "hits.hits" } to the dataset, that helped to reduce the elasticsearch output significantly, but still didn't work fully. I need to do some more due diligence on this and then if needed I will add details here (or maybe in a separate thread).

Thanks.


#5

Hi @nyuriks and @rashmi,

Looks like I have been able to reach beyond these two problems now.

For json - I had to save it in raw format on gist.github.com, and then it worked. I haven't actually figured out how it returns the data differently if saved in raw or otherwise, but I decided to move on.

For index - apart from the "format": { "property": "hits.hits" }, I had to refer to my fields as "_source.<field_name>". The VEGADEBUG.view.data() has been an invaluable help.

Thanks so much.


(Tony Skulk) #6

Hi,

I´m starting to build my own vega-lite visualizations in kibana but I´m a little stuck. I think my problem is very much related to yours so I hope it´s ok to reply on your thread.

I´ve just built an index and filled it with over 5M entries. But I can´t manage to get any data to be shown on my plot. Here is my vega-lite spec:

{
  $schema: https://vega.github.io/schema/vega-lite/v2.json
  data: {
    name: healthData
    url: {
      %context%: true
      %timefield%: @timestamp
      index: gaptrader.heartbeat
      body: {
        size: 1000
      }
    }
    format: { property: "hits.hits" }
  }
  layer: [
    {
      mark: {type: "line", interpolate: "step-after"}
      encoding: {
        x: {
          field: timestamp
          type: temporal
          timeUnit: hoursminutesseconds
          axis: {grid: false, ticks: false, title: null}
        }
        y: {
          field: lastSignalSeconds
          type: quantitative
          scale: {
            domain: [0, 600]
            type: sqrt
          }
          axis: {grid: false, ticks: false, title: null}
        }
      }
    }
    {
      mark: line
      encoding: {
        x: {
          field: timestamp
          type: temporal
          timeUnit: hoursminutesseconds
          axis: {grid: false, ticks: false, title: null}
        }
        y: {field: "warning", type: "quantitative"}
        color: {value: "#f2c037"}
        opacity: {value: "0.5"}
      }
    }
    {
      mark: line
      encoding: {
        x: {
          field: timestamp
          type: temporal
          timeUnit: hoursminutesseconds
          axis: {grid: false, ticks: false, title: null}
        }
        y: {field: "error", type: "quantitative"}
        color: {value: "#db2828"}
        opacity: {value: "0.3"}
      }
    }
  ]
}

When I am looking at the network request Kibana sends I see that Kibana does not return any hits. What am I doing wrong?

Request-Body:

{"size":1000,"query":{"bool":{"must":[{"match_all":{}},{"range":{"@timestamp":{"gte":1380387787229,"lte":1538154187229,"format":"epoch_millis"}}}],"must_not":[]}}}

Response:

{"took":0,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":0,"max_score":null,"hits":[]}}

Maybe there is a basic mistake I made, writing the "data"-spec in Kibana. But I can not find out why I get empty hits.


#7

@tonyskulk, I am not sure off hand, but one thing might be worth looking at is to remove the %context%: true specification. Your query seems to be looking for data where timestamp is between "Saturday, September 28, 2013 5:03:07.229 PM" (GMT) and "Friday, September 28, 2018 5:03:07.229 PM" (GMT) [Ref: https://www.epochconverter.com/]. Is it possible that your index doesn't have a matching data for that range?

-- Parag


(Tony Skulk) #8

Thank you @deuskars for your answer.
I tried different filter specs like last 5 years with or without %context%: true but could not get any data shown in my visualization.

I at least found something: When I use

index: *

then in the corresponding http request´s (.../elasticsearch/*/_search) response, there are over 1 million hits. Unfortunalety none of my gaptrader.heartbeat data, but at least all hits regarding to some filebeat*-indexes.

So I think it has something to do with my vega-lite specification on the data.url.index attribute.

Currently I am using

index: gaptrader.heartbeat

and nothing has been returned. But when I run the same query in Kibana Dev Tools

GET gaptrader.streamgap/_search

I got over 5 million data points returned. So there must be a difference between dev tools and the /_search endpoint...
Does anyone has also faced that problem?


(Tony Skulk) #9

Ok, I found the mistake. I used

%timefield%: @timestamp

Instead using

%timefield%: timestamp

results in the desired hits.


(system) #10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.