Accessing Elasticsearch from Vega to create custom visualizations

[EricJohnson]

Greetings,

I bet this problem will be easy for someone out there - but I can't see what I'm doing wrong. I have data in an Elasticsearch index called "plotting_test" that I'd like to plot with Vega. I'm looking at the blog entry https://www.elastic.co/blog/custom-vega-visualizations-in-kibana for reference.

All of the documents in the index look like this:

{
  "_index": "plotting_test",
  "_type": "text",
  "_id": "CgNrA20BpOHJNpjkfGzX",
  "_version": 1,
  "_score": 0,
  "_source": {
    "time": "2019-08-14",
    "ds": "2019-08-14T00:00:00",
    "data": 0.4997,
    "yhat": 0.5541273960954143,
    "yhat_lower": 0.2884054750552231,
    "yhat_upper": 0.8083939855874178
   },
  "fields": {
    "ds": [
      "2019-08-14T00:00:00.000Z"
 ],
  "time": [
      "2019-08-14T00:00:00.000Z"
    ]
  }
}

I'm trying to graph x = ds and y = yhat. This is the Vega code I have so far:

{
  $schema: https://vega.github.io/schema/vega-lite/v2.json
  data: {
    url: {
      %context%: true
      %timefield%: "ds"
      index: plotting_test
      body: {
        size: 10000
        _source: ["ds", "yhat"]
      }
    }
    # We only need the content of hits.hits array
    format: {property: "hits.hits"}
  }
  # Parse timestamp into a javascript date value
  transform: [
    {calculate: "toDate(datum._source['ds'])", as: "time"}
  ]
  # Draw a circle, with x being the time field, and y - number of bytes
  mark: circle
  encoding: {
    x: {field: "ds", type: "temporal"}
    y: {field: "yhat", type: "quantitative"}
  }
}

Using the Kibana dev tool I can get the data no problem by simply writing

GET /plotting_test/_search
{
  "query": {
      "match_all": {}
  },
  "_source":["yhat", "ds"]
}

I can't figure out what is wrong with the Vega code. If I can just get one example to work locally I can start doing all of my analytics plots in Vega.

Thank you!

[wylie (Wylie Conlon)]

I think your XY encoding is not matching any fields. Because of the document shape you showed earlier, you still need _source as a prefix. Also, I see that you are using the transform clause but not actually encoding it. Putting it together, here is what I think you could try:

  encoding: {
    x: {field: "time", type: "temporal"}
    y: {field: "_source.yhat", type: "quantitative"}
  }

I only changed the X and Y encodings from your previous example.

[EricJohnson]

Thank you so much! It worked!

Yay!