How to plot (raw) data, Using Vega

Hey @defalt. I think that the format: { property: "hits.hits" } is missing on data field. If you hit on the Dev Console of kibana the:

GET modbusdata_new/_search 
    {
      "size": 10000,
      "_source": ["timestamp", "T107","C106"]
}

You will get a response like that:

"hits" : {
    "total" : {
      "value" : 5,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "example",
        "_id" : "1",
        "_score" : 1.0,
        "_source" : {
          "timestamp" : "A",
          "T107" : 8,
          "C106": 9
        }
      },

The path to _source is hits.hits so you should also include it. Your data should be like:

  data: {
    url: {
      %context%: true
      %timefield%: timestamp
      index: kibana_sample_data_logs
      body: {
        size: 10000
        _source: ["timestamp", "T107","C106"]
      }
    }
    format: {
      property: hits.hits
    }
  }

So now you will be able to get the info as _source.T107

Hope it helps :slight_smile:

Stratoula

This helped a lot @Stratoula_Kalafateli . Now I can see data. But the problem is the data isn't drawn accross the X-Axis. Its drawn in a single line (All timestamps are in a single point I guess). I couldn't find a solution to this.

{
  "$schema": "https://vega.github.io/schema/vega/v4.3.json",
  "description": "A basic line chart example.",
  "padding": 5,
  
  data: {
  name:"table"
    url: {
      %context%: true
      %timefield%: timestamp
      index: modbusdata_new
      body: {
        "size": 10000,
        "_source": ["timestamp", "T107","C106"]
      },
    }
    format: { property: "hits.hits" }
  }
    transform: [
    {
      calculate: "toDate(datum._source['timestamp'])"
      as: "time"
    }
  ],
  
  "scales": [
    {
      "name": "timex",
      "type": "time",
      "range": "width",
      "domain": {"data": "table", "field": "time"}
    },
    {
      "name": "Namet",
      "type": "linear",
      "range": "height",
      "nice": true,
      "zero": true,
      "domain": {"data": "table", "field": "_source.T107"}
    },
    {
      "name": "Namec",
      "type": "linear",
      "range": "height",
      "nice": true,
      "zero": true,
      "domain": {"data": "table", "field": "_source.C106"}
    }
  ],
  "axes": [
    {"orient": "bottom", "scale": "timex", "format": "%d:%m:%H:%m"},
    {"orient": "left", "scale": "Namet"},
    {
      "orient": "left",
      "scale": "Namec",
      "offset": 30,
      "domainColor": "#FF00FF",
      "labelColor": "#FF00FF",
      "tickColor": "#FF00FF"
    }
  ],
  "marks": [
  {
      "type": "line",
      "from": {"data": "table"},
      "encode": {
        "enter": {
          "x": {"scale": "timex", "field": "time"},
          "y": {"scale": "Namec", "field": "_source.C106"},
          "stroke": {"value": "#FF00FF"}
        }
      }
    },
    {
      "type": "line",
      "from": {"data": "table"},
      "encode": {
        "enter": {
          "x": {"scale": "timex", "field": "time"},
          "y": {"scale": "Namet", "field": "_source.T107"}
        }
      }
    }
  ]
}

Output:

When I change the timescale the data changes too. So its some type of a time error.
Thanks for your help.

Hey @defalt! I assume that the problem here is that as we are using Vega now (instead of vega-lite) the "transform" pipeline now should appear on data. You also need to replace "calculate" with the Vega "formula".
So it should be something like:

data: {
  name:"table"
    url: {
      %context%: true
      %timefield%: timestamp
      index: modbusdata_new
      body: {
        "size": 10000,
        "_source": ["timestamp", "T107","C106"]
      },
    }
    format: { property: "hits.hits" },
    transform: [
        { type: "formula",as: "time", expr: "datetime(datum._source.timestamp) "}
      ]
  }

Also check the supported date functions here.

Awesome, it works now. Is there a way to display data from different indices too? If I write index: index1,index2 its really buggy and doesn't display the right things.
Lets say I have Index 1 with T107 and C106 and Index 2 with O145. Is there a way to display them in the same graph?

Yes you can! data can be an array on vega so you could have:

data: [
{
  name:"table"
    url: {
      %context%: true
      %timefield%: timestamp
      index: modbusdata_new
      body: {
        "size": 10000,
        "_source": ["timestamp", "T107","C106"]
      },
    }
    format: { property: "hits.hits" },
    transform: [
        { type: "formula",as: "time", expr: "datetime(datum._source.timestamp) "}
      ]
  },
{
  name:"table2"
    url: {
      %context%: true
      %timefield%: timestamp
      index: my_second_index
      body: {
        "size": 10000,
        "_source": ["timestamp", "O145"]
      },
    }
    format: { property: "hits.hits" },
    transform: [
        { type: "formula",as: "time", expr: "datetime(datum._source.timestamp) "}
      ]
  }
]

Then you could add a new scale with domain the new dataset "domain": {"data": "table2", "field": "_source.O145"}, a new axis and a new mark:

{
      "type": "line",
      "from": {"data": "table2"},
      "encode": {
        "enter": {
          "x": {"scale": "x", "field": "time"},
          "y": {"scale": "y2", "field": "_source.O145"},
          "stroke": {"value": "#FF00FF"}
        }
      }
    }

Thanks for all your help @Stratoula_Kalafateli . Do you have an idea why my timing scale is weird? I don't think it displays correct data and it doesn't change with the timeframe set in kibana. It jumps from 11:05 to 12:05

"axes": [
    {"orient": "bottom", "scale": "timex", "format": "%H:%m","grid":"true"},

Hey @defalt :slight_smile: I can't imagine why this happens. Do you want to copy paste your configuration? Maybe it rings a bell.

Config

{
  "$schema": "https://vega.github.io/schema/vega/v4.3.json",
  "description": "A basic line chart example.",
  "padding": 5,
  
data: [
{
  name:"table"
    url: {
      %context%: true
      %timefield%: @timestamp
      index: index1
      body: {
        "size": 17000,
        "_source": ["@timestamp", "value1","value2","value3"],
        "sort" : { "@timestamp" : "desc" }
      },
    }
    format: { property: "hits.hits" },
    transform: [
        { type: "formula",as: "time", expr: "datetime(datum._source['@timestamp']) "}
      ]
  },
{
  name:"table2"
    url: {
      %context%: true
      %timefield%: @timestamp
      index: index2
      body: {
        "size": 17000,
        "_source": ["@timestamp", "value4"],
        "sort" : { "@timestamp" : "desc" }
      },
    }
    format: { property: "hits.hits" },
    transform: [
        { type: "formula",as: "time2", expr: "datetime(datum._source['@timestamp']) "}
      ]
  }
]
  "scales": [
    {
      "name": "timex",
      "type": "time",
      "range": "width",
      "domain": {"data": "table", "field": "time"}
    },
    {
      "name": "Namet",
      "type": "linear",
      "range": "height",
      "nice": true,
      "zero": true,
      "domain": {"data": "table", "field": "_source.value1"}
    },
    {
      "name": "Namec",
      "type": "linear",
      "range": "height",
      "nice": true,
      "zero": true,
      "domain": {"data": "table", "field": "_source.value2"}
    },
    {
      "name": "Namevalue3",
      "type": "linear",
      "range": "height",
      "nice": true,
      "zero": true,
      "domain": {"data": "table", "field": "_source.value3"}
    },
    {
      "name": "NameValue4",
      "type": "linear",
      "range": "height",
      "nice": true,
      "zero": true,
      "domain": {"data": "table2", "field": "_source.value4"}
    }
  ],
  "axes": [
    {
    "orient": "bottom", 
    "scale": "timex", 
    "format": "%s",
    "grid":"true"},
    {
    "orient": "left", 
    "scale": "Namet",
    "grid":"true"},
    {
      "orient": "left",
      "scale": "Namec",
      "offset": 30,
      "domainColor": "#FF00FF",
      "labelColor": "#FF00FF",
      "tickColor": "#FF00FF",
      "grid":"true"
    },
    {
      "orient": "left",
      "scale": "NameValue4",
      "offset": 60,
      "domainColor": "#CCCCCC",
      "labelColor": "#CCCCCC",
      "tickColor": "#CCCCCC",
      "grid":"true"
    },
    {
      "orient": "left",
      "scale": "Namevalue3",
      "offset": 90,
      "domainColor": "#444444",
      "labelColor": "#444444",
      "tickColor": "#444444",
      "grid":"true"
    }
  ],
  "marks": [
  {
      "type": "line",
      "from": {"data": "table"},
      "encode": {
        "enter": {
          "x": {"scale": "timex", "field": "time"},
          "y": {"scale": "Namec", "field": "_source.value2"},
          "stroke": {"value": "#FF00FF"}
        }
      }
    },
    {
      "type": "line",
      "from": {"data": "table"},
      "encode": {
        "enter": {
          "x": {"scale": "timex", "field": "time"},
          "y": {"scale": "Namet", "field": "_source.value1"}
        }
      }
    },
    {
      "type": "line",
      "from": {"data": "table2"},
      "encode": {
        "enter": {
          "x": {"scale": "timex", "field": "time2"},
          "y": {
          "scale": "NameValue4",
          "field": "_source.value4"
          },
          "stroke": {"value": "#CCCCCC"}
        }
      }
    },
    {
      "type": "line",
      "from": {"data": "table"},
      "encode": {
        "enter": {
          "x": {"scale": "timex", "field": "time"},
          "y": {
          "scale": "Namevalue3", "field": "_source.value3"},
          "stroke": {"value": "#444444"}
        }
      }
    }
  ]
}

This is the scale in %s, seconds.


According to the scale i woud insert data in nanoseconds. But thats not the case:

Here you can see what the timescale should look like.

ok, I see that you set time and time2. I would use the same variable time.

So this is an example configuration with two lines (I have used the kibana sample datasets)

{
  "$schema": "https://vega.github.io/schema/vega/v4.3.json",
  "description": "A basic line chart example.",
  "padding": 5,
  "data": [
    {
  	// query ES based on the currently selected time range and filter string
  	name: table
  	url: {
    	%context%: true
    	%timefield%: timestamp
    	index: kibana_sample_data_flights
    	body: {
        "size": 10000,
        "_source": ["timestamp", "DistanceKilometers"],
        "sort" : { "timestamp" : "desc" }
      }
  	}
  	// From the result, take just the data we are interested in
  	format: {property: "hits.hits"}
  	transform: [
    	{type: "formula", expr: "datetime(datum._source.timestamp)", as: "time"}
  	]
	},
	{
  	// query ES based on the currently selected time range and filter string
  	name: table2
  	url: {
    	%context%: true
    	%timefield%: timestamp
    	index: kibana_sample_data_logs
    	body: {
        "size": 10000,
        "_source": ["timestamp", "bytes"],
         "sort" : { "timestamp" : "desc" }
      }
  	}
  	// From the result, take just the data we are interested in
  	format: {property: "hits.hits"}
  	transform: [
    	{type: "formula", expr: "datetime(datum._source.timestamp)", as: "time"}
  	]
	}
  ],
  "scales": [
    {
      "name": "x",
      "type": "time",
      "range": "width",
      "domain": {"data": "table", "field": "time"}
    },
    {
      "name": "y",
      "type": "linear",
      "range": "height",
      "nice": true,
      "zero": true,
      "domain": {"data": "table", "field": "_source.DistanceKilometers"}
    },
    {
      "name": "y2",
      "type": "linear",
      "range": "height",
      "nice": true,
      "zero": true,
      "domain": {"data": "table2", "field": "_source.bytes"}
    }
  ],
  "axes": [
    {"orient": "bottom", "scale": "x", "grid": true, "format": "%s",},
    {"orient": "left", "scale": "y"},
    {
      "orient": "left",
      "scale": "y2",
      "offset": 50,
      "domainColor": "#FF00FF",
      "labelColor": "#FF00FF",
      "tickColor": "#FF00FF"
    }
  ],
  "marks": [
    {
      "type": "line",
      "from": {"data": "table"},
      "encode": {
        "enter": {
          "x": {"scale": "x", "field": "time"},
          "y": {"scale": "y", "field": "_source.DistanceKilometers"}
        }
      }
    },
    {
      "type": "line",
      "from": {"data": "table2"},
      "encode": {
        "enter": {
          "x": {"scale": "x", "field": "time"},
          "y": {"scale": "y2", "field": "_source.bytes"},
          "stroke": {"value": "#FF00FF"}
        }
      }
    }
  ]
}

The above example changes with the kibana timepicker and the time scale seems to work properly.

Replaced time2 with time. Still the same error tho. But thanks for the try. Any ideas on how to debug this?

If you remove completely the line from the second index, does it work correctly ? I would try these combinations to understand when the problem is introduced. Also check my example, I think it will help. Finally, also check your datasets, the @timestamp field.

  1. If I remove transform: [ { type: "formula",as: "time", expr: "datetime(datum._source['@timestamp']) "} ] the grey line is gone. It needs the timeframe to be drawn.
  2. I think there is no real difference between your example and my code. I always tried to stay close to the code you and tim gave me.
  3. The Timestamps look like this.
{
  "_index": "index",
  "_type": "index",
  "_id": "AXAJ3VEaaIwsdfT2yAntx",
  "_version": 1,
  "_score": null,
  "_source": {
   "valueXY": XYZ
    "@timestamp": "2020-02-02T08:17:47.959Z",
  },
  "fields": {
    "@timestamp": [
      "2020-02-02T08:17:47.959Z"
    ],
    "timestamp": [
      "2020-02-02T09:17:47.959Z"
    ]
  },
}

If I change my code from @timestamp to timestamp I get "Cannot read property 'hits' of undefined".

My suggestion was to remove temporarily the one line from the second index in order to check if the other three lines from the first index are displayed correctly (for debugging purposes).

Something like that:

{
  "$schema": "https://vega.github.io/schema/vega/v4.3.json",
  "description": "A basic line chart example.",
  "padding": 5,
  
data: [
{
  name:"table"
    url: {
      %context%: true
      %timefield%: @timestamp
      index: index1
      body: {
        "size": 17000,
        "_source": ["@timestamp", "value1","value2","value3"],
        "sort" : { "@timestamp" : "desc" }
      },
    }
    format: { property: "hits.hits" },
    transform: [
        { type: "formula",as: "time", expr: "datetime(datum._source['@timestamp']) "}
      ]
  }
]
  "scales": [
    {
      "name": "timex",
      "type": "time",
      "range": "width",
      "domain": {"data": "table", "field": "time"}
    },
    {
      "name": "Namet",
      "type": "linear",
      "range": "height",
      "nice": true,
      "zero": true,
      "domain": {"data": "table", "field": "_source.value1"}
    },
    {
      "name": "Namec",
      "type": "linear",
      "range": "height",
      "nice": true,
      "zero": true,
      "domain": {"data": "table", "field": "_source.value2"}
    },
    {
      "name": "Namevalue3",
      "type": "linear",
      "range": "height",
      "nice": true,
      "zero": true,
      "domain": {"data": "table", "field": "_source.value3"}
    }
  ],
  "axes": [
    {
    "orient": "bottom", 
    "scale": "timex", 
    "format": "%s",
    "grid":"true"},
    {
    "orient": "left", 
    "scale": "Namet",
    "grid":"true"},
    {
      "orient": "left",
      "scale": "Namec",
      "offset": 30,
      "domainColor": "#FF00FF",
      "labelColor": "#FF00FF",
      "tickColor": "#FF00FF",
      "grid":"true"
    },
    {
      "orient": "left",
      "scale": "Namevalue3",
      "offset": 90,
      "domainColor": "#444444",
      "labelColor": "#444444",
      "tickColor": "#444444",
      "grid":"true"
    }
  ],
  "marks": [
  {
      "type": "line",
      "from": {"data": "table"},
      "encode": {
        "enter": {
          "x": {"scale": "timex", "field": "time"},
          "y": {"scale": "Namec", "field": "_source.value2"},
          "stroke": {"value": "#FF00FF"}
        }
      }
    },
    {
      "type": "line",
      "from": {"data": "table"},
      "encode": {
        "enter": {
          "x": {"scale": "timex", "field": "time"},
          "y": {"scale": "Namet", "field": "_source.value1"}
        }
      }
    },
    {
      "type": "line",
      "from": {"data": "table"},
      "encode": {
        "enter": {
          "x": {"scale": "timex", "field": "time"},
          "y": {
          "scale": "Namevalue3", "field": "_source.value3"},
          "stroke": {"value": "#444444"}
        }
      }
    }
  ]
}

So if the three lines are displayed correctly, the error is on the 4th line.
@timestamp seems fine to me.

Got rid of table 2, still the same problem.

Edit:
I will try to solve this "the hard way". I will create a new index with sample data. If the error stays the same the code is faulty, if the error is gone, my data is in some way faulty.

@timroes are there any plans to make raw data visualization a standard feature in kibana? I think everyone who uses this for measuring voltages etc. will thank you for it :wink:

1 Like

Sounds good, looking forward to the results :slight_smile:

The same problem occurs with different data. When I zoomed out it looks like everything is correct and the time fits but when the timescale is smaller the same error appears.


{
  "$schema": "https://vega.github.io/schema/vega/v4.3.json",
  "description": "A basic line chart example.",
  "padding": 5,
  
data: [
{
  name:"table"
    url: {
      %context%: true
      %timefield%: @timestamp
      index: elasticoffee
      body: {
        "size": 10000,
        "_source": ["@timestamp", "quadId"],
        "sort" : { "@timestamp" : "desc" }
      },
    }
    format: { property: "hits.hits" },
    transform: [
        { type: "formula",as: "time", expr: "datetime(datum._source['@timestamp'])"}
      ]
  }
],
  "scales": [
    {
      "name": "timex",
      "type": "time",
      "range": "width",
      "domain": {"data": "table", "field": "time"}
    },
    {
      "name": "NameBeverage",
      "type": "linear",
      "range": "height",
      "nice": true,
      "zero": true,
      "domain": {"data": "table", "field": "_source.quadId"}
    }
  ],
  "axes": [
    {
    "orient": "bottom", 
    "scale": "timex", 
    "format": "%d %H:%m",
    "grid":"true"
    },
    {
    "orient": "left", 
    "scale": "NameBeverage",
    "grid":"true"},
  ],
  "marks": [
  {
      "type": "line",
      "from": {"data": "table"},
      "encode": {
        "enter": {
          "x": {"scale": "timex", "field": "time"},
          "y": {"scale": "NameBeverage", "field": "_source.quadId"},
          "stroke": {"value": "#FF00FF"},
          "strokeWidth": "1"
        }
      }
    }
  ]
}

The used data was from Elastic Coffee and looks something like this:

POST _bulk
{ "index" : { "_index" : "elasticoffee", "_type" : "doc", "_id" : "5005" } }
{"sceneID": "2", "sceneData": "0", "entityID": "zwave.quad2", "quadId": 15, "quadMod": "1", "@timestamp": "2018-08-30T08:26:39Z", "beverageClass": "Hot Beverages", "beverage": "Latte", "beverageSide": "left", "beverageIndex": 5, "quantity": 1}

I used quadId for my testing purpose. I am clueless why this time shift happens. May best guess is that is has something to do with the transform function, its the only thing that can manipulate the time.

Thank you @defalt for pointing this out! Will look into this, I agree transform function may be the reason.

Oh my... Found it.
"format": "%d %H:%m" or %s
is wrong. %m = Month and %s is unix epoch (Link)
Changed it to "format": "%d %H:%M,%S" and it looks good now.

Indeed, I was testing it with %H:%M. Great, I am really happy that works for you now :slight_smile: :clap:

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.