Can't Get Timestamp Transform To Work For A Basic Vega Scatter Plot

Elastic Stack Kibana
1

Here is my Vega code

{
  "$schema": "https://vega.github.io/schema/vega/v4.json",
  "description": "A basic scatter plot",
  "padding": 5,

  "data": {
    "name": "source",
    "url": {
      "%context%": "true",
      "%timefield%": "@timestamp",
      "index": "alb-logs-*",
      "body": {
        "size": 1000,
        "_source": ["@timestamp", "elb_received_bytes"],
      }
    },
    "format": {"property": "hits.hits"},
  },
  "transform": [
      {
        calculate: "toDate(datum._source['@timestamp'])" as: "time"
      }
    ]
  "scales": [
    {
      "name": "x",
      "type": "time",
      "round": true,
      "nice": true,
      "zero": true,
      "domain": {"data": "source", "field": "time"},
      "range": "width"
    },
    {
      "name": "y",
      "type": "linear",
      "round": true,
      "nice": true,
      "zero": true,
      "domain": {"data": "source", "field": "_source.elb_received_bytes"},
      "range": "height"
    }
  ],

  "axes": [
    {
      "scale": "x",
      "grid": true,
      "domain": false,
      "orient": "bottom",
      "tickCount": 5,
      "title": "Time"
    },
    {
      "scale": "y",
      "grid": true,
      "domain": false,
      "orient": "left",
      "titlePadding": 5,
      "title": "Bytes"
    }
  ],

  

  "marks": [
    {
      "name": "marks",
      "type": "symbol",
      "from": {"data": "source"},
      "encode": {
        "update": {
          "x": {"scale": "x", "field": "time"},
          "y": {"scale": "y", "field": "_source.elb_received_bytes"},
          "shape": {"value": "circle"},
          "strokeWidth": {"value": 2},
          "opacity": {"value": 0.5},
          "stroke": {"value": "#4682b4"},
          "fill": {"value": "transparent"}
        }
      }
    }
  ]
}

When I run the code, my time data is missing and I am seeing following output:

8279db38-577f-449e-9181-6e92b1d6b42d
8279db38-577f-449e-9181-6e92b1d6b42d957×885 32.7 KB

IN my web console, when I run VEGA_DEBUG.view.data('source'), I don't see the "time" field in my data array. So, I believe that my transform function is not working properly.
Here is what I see in my web console

0:
Symbol(vega_id): 3935923
_id: "Man2mHcBbBSZp3usZgN-"
_index: "alb-logs-2020.12"
_score: 1
_source:
@timestamp: "2020-12-16T20:00:03.303Z"
elb_received_bytes: "241"
2

OK. I made some progress. Apparently, I was using Vega-lite expression for transform. I changed the expression and made it vega specific :

"transform": [
      {
        "type": "formula", "as": "time", "expr": "toDate(datum._source['@timestamp'])" 
      }
    ]

and now it appears to be working but it led to another problem as shown below

image
image1182×908 39.7 KB

As you can see, the x-axis is not scaling automatically according to the timestamp range of the data points.

Also, why can't I still see the transformed field "time" values in chrome console? I only see the 2 original elastic index fields as mentioned in my previous comment. How does one see the transformed fields in browser console?

3

Could you provide the spec?

4

@aaron-nimocks thanks for your response. Here are the specs. I have reduced the number of data points to 10 so that it can be easily accommodated here. The problem still persists even with 10 data points.

{
  "$schema": "https://vega.github.io/schema/vega/v4.json",
  "description": "A basic scatter plot",
  "padding": 5,
  "data": {
    "name": "source",
    "format": {
      "property": "hits.hits"
    },
    "transform": [
      {
        "type": "formula",
        "as": "time",
        "expr": "toDate(datum._source['@timestamp'])"
      }
    ],
    "url": {
      "index": "alb-logs-*",
      "body": {
        "size": 10,
        "_source": [
          "@timestamp",
          "elb_received_bytes"
        ],
        "query": {
          "bool": {
            "must": [
              {
                "range": {
                  "@timestamp": {
                    "gte": "2021-02-18T00:47:15.704Z",
                    "lte": "2021-02-18T01:02:15.704Z",
                    "format": "strict_date_optional_time"
                  }
                }
              }
            ],
            "filter": [
              {
                "match_all": {}
              }
            ],
            "should": [],
            "must_not": []
          }
        }
      }
    },
    "values": {
      "took": 6,
      "timed_out": false,
      "_shards": {
        "total": 3,
        "successful": 3,
        "skipped": 2,
        "failed": 0
      },
      "hits": {
        "total": 1487,
        "max_score": 1,
        "hits": [
          {
            "_index": "alb-logs-2021.02",
            "_type": "_doc",
            "_id": "6ViesncBHSONY29jDXEY",
            "_score": 1,
            "_source": {
              "@timestamp": "2021-02-18T00:47:36.638Z",
              "elb_received_bytes": "236"
            },
            "time": 1613609256638
          },
          {
            "_index": "alb-logs-2021.02",
            "_type": "_doc",
            "_id": "6liesncBHSONY29jDXEY",
            "_score": 1,
            "_source": {
              "@timestamp": "2021-02-18T00:48:04.753Z",
              "elb_received_bytes": "236"
            },
            "time": 1613609284753
          },
          {
            "_index": "alb-logs-2021.02",
            "_type": "_doc",
            "_id": "61iesncBHSONY29jDXEY",
            "_score": 1,
            "_source": {
              "@timestamp": "2021-02-18T00:47:23.721Z",
              "elb_received_bytes": "202"
            },
            "time": 1613609243721
          },
          {
            "_index": "alb-logs-2021.02",
            "_type": "_doc",
            "_id": "7FiesncBHSONY29jDXEY",
            "_score": 1,
            "_source": {
              "@timestamp": "2021-02-18T00:47:48.237Z",
              "elb_received_bytes": "449"
            },
            "time": 1613609268237
          },
          {
            "_index": "alb-logs-2021.02",
            "_type": "_doc",
            "_id": "7ViesncBHSONY29jDXEY",
            "_score": 1,
            "_source": {
              "@timestamp": "2021-02-18T00:47:53.094Z",
              "elb_received_bytes": "241"
            },
            "time": 1613609273094
          },
          {
            "_index": "alb-logs-2021.02",
            "_type": "_doc",
            "_id": "7liesncBHSONY29jDXEY",
            "_score": 1,
            "_source": {
              "@timestamp": "2021-02-18T00:47:59.482Z",
              "elb_received_bytes": "69"
            },
            "time": 1613609279482
          },
          {
            "_index": "alb-logs-2021.02",
            "_type": "_doc",
            "_id": "71iesncBHSONY29jDXEY",
            "_score": 1,
            "_source": {
              "@timestamp": "2021-02-18T00:48:11.755Z",
              "elb_received_bytes": "165"
            },
            "time": 1613609291755
          },
          {
            "_index": "alb-logs-2021.02",
            "_type": "_doc",
            "_id": "8FiesncBHSONY29jDXEY",
            "_score": 1,
            "_source": {
              "@timestamp": "2021-02-18T00:48:20.981Z",
              "elb_received_bytes": "1373"
            },
            "time": 1613609300981
          },
          {
            "_index": "alb-logs-2021.02",
            "_type": "_doc",
            "_id": "8ViesncBHSONY29jDXEY",
            "_score": 1,
            "_source": {
              "@timestamp": "2021-02-18T00:48:20.258Z",
              "elb_received_bytes": "161"
            },
            "time": 1613609300258
          },
          {
            "_index": "alb-logs-2021.02",
            "_type": "_doc",
            "_id": "8liesncBHSONY29jDXEY",
            "_score": 1,
            "_source": {
              "@timestamp": "2021-02-18T00:48:21.460Z",
              "elb_received_bytes": "90"
            },
            "time": 1613609301460
          }
        ]
      }
    }
  },
  "scales": [
    {
      "name": "x",
      "type": "time",
      "round": true,
      "nice": true,
      "zero": true,
      "domain": {
        "data": "source",
        "field": "time"
      },
      "range": "width"
    },
    {
      "name": "y",
      "type": "linear",
      "round": true,
      "nice": true,
      "zero": true,
      "domain": {
        "data": "source",
        "field": "_source.elb_received_bytes"
      },
      "range": "height"
    }
  ],
  "axes": [
    {
      "scale": "x",
      "grid": true,
      "domain": false,
      "orient": "bottom",
      "tickCount": 5,
      "title": "Time"
    },
    {
      "scale": "y",
      "grid": true,
      "domain": false,
      "orient": "left",
      "titlePadding": 5,
      "title": "Bytes"
    }
  ],
  "marks": [
    {
      "name": "marks",
      "type": "symbol",
      "from": {
        "data": "source"
      },
      "encode": {
        "update": {
          "x": {
            "scale": "x",
            "field": "time"
          },
          "y": {
            "scale": "y",
            "field": "_source.elb_received_bytes"
          },
          "shape": {
            "value": "circle"
          },
          "strokeWidth": {
            "value": 2
          },
          "opacity": {
            "value": 0.5
          },
          "stroke": {
            "value": "#4682b4"
          },
          "fill": {
            "value": "transparent"
          }
        }
      }
    }
  ],
  "config": {
    "range": {
      "category": {
        "scheme": "elastic"
      }
    },
    "arc": {
      "fill": "#54B399"
    },
    "area": {
      "fill": "#54B399"
    },
    "line": {
      "stroke": "#54B399"
    },
    "path": {
      "stroke": "#54B399"
    },
    "rect": {
      "fill": "#54B399"
    },
    "rule": {
      "stroke": "#54B399"
    },
    "shape": {
      "stroke": "#54B399"
    },
    "symbol": {
      "fill": "#54B399"
    },
    "trail": {
      "fill": "#54B399"
    },
    "title": {
      "color": "#343741"
    },
    "style": {
      "guide-label": {
        "fill": "#69707d"
      },
      "guide-title": {
        "fill": "#343741"
      },
      "group-title": {
        "fill": "#343741"
      },
      "group-subtitle": {
        "fill": "#343741"
      }
    },
    "axis": {
      "tickColor": "#eef0f3",
      "domainColor": "#eef0f3",
      "gridColor": "#eef0f3"
    },
    "background": "transparent"
  },
  "width": "container",
  "height": "container",
  "autosize": {
    "type": "fit",
    "contains": "padding"
  }
}
5

There is a new aggregate called timeunit but it's in V5 and here is the example.

I made all the changes here to make that work. Let me know if this all makes sense.

6

Got it to work. Thanks for the solution :slight_smile:

image
image1173×1000 181 KB

1 Like
7

I am just seeking one more clarification. What is the purpose of following transform?

{
          "type": "aggregate",
          "groupby": ["unit0"],
          "ops": ["average"],
          "fields": ["_source.elb_received_bytes"],
          "as": ["elb_received_bytes"]
        }

When I insert this into my code, I get an error Cannot read property 'elb_received_bytes' of undefined

8

That one groups the unit0 field that was created with the timeunit aggregation and adds in your aggregation for your y-axis.

So if in timeunit you specify minutes then that aggregation will take the average bytes and group them by unit0 so every minute is accounted for and you can create the chart.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.