Can't Get Timestamp Transform To Work For A Basic Vega Scatter Plot

Here is my Vega code

{
  "$schema": "https://vega.github.io/schema/vega/v4.json",
  "description": "A basic scatter plot",
  "padding": 5,

  "data": {
    "name": "source",
    "url": {
      "%context%": "true",
      "%timefield%": "@timestamp",
      "index": "alb-logs-*",
      "body": {
        "size": 1000,
        "_source": ["@timestamp", "elb_received_bytes"],
      }
    },
    "format": {"property": "hits.hits"},
  },
  "transform": [
      {
        calculate: "toDate(datum._source['@timestamp'])" as: "time"
      }
    ]
  "scales": [
    {
      "name": "x",
      "type": "time",
      "round": true,
      "nice": true,
      "zero": true,
      "domain": {"data": "source", "field": "time"},
      "range": "width"
    },
    {
      "name": "y",
      "type": "linear",
      "round": true,
      "nice": true,
      "zero": true,
      "domain": {"data": "source", "field": "_source.elb_received_bytes"},
      "range": "height"
    }
  ],

  "axes": [
    {
      "scale": "x",
      "grid": true,
      "domain": false,
      "orient": "bottom",
      "tickCount": 5,
      "title": "Time"
    },
    {
      "scale": "y",
      "grid": true,
      "domain": false,
      "orient": "left",
      "titlePadding": 5,
      "title": "Bytes"
    }
  ],

  

  "marks": [
    {
      "name": "marks",
      "type": "symbol",
      "from": {"data": "source"},
      "encode": {
        "update": {
          "x": {"scale": "x", "field": "time"},
          "y": {"scale": "y", "field": "_source.elb_received_bytes"},
          "shape": {"value": "circle"},
          "strokeWidth": {"value": 2},
          "opacity": {"value": 0.5},
          "stroke": {"value": "#4682b4"},
          "fill": {"value": "transparent"}
        }
      }
    }
  ]
}

When I run the code, my time data is missing and I am seeing following output:

IN my web console, when I run VEGA_DEBUG.view.data('source'), I don't see the "time" field in my data array. So, I believe that my transform function is not working properly.
Here is what I see in my web console

0:
Symbol(vega_id): 3935923
_id: "Man2mHcBbBSZp3usZgN-"
_index: "alb-logs-2020.12"
_score: 1
_source:
@timestamp: "2020-12-16T20:00:03.303Z"
elb_received_bytes: "241"

OK. I made some progress. Apparently, I was using Vega-lite expression for transform. I changed the expression and made it vega specific :

"transform": [
      {
        "type": "formula", "as": "time", "expr": "toDate(datum._source['@timestamp'])" 
      }
    ]

and now it appears to be working but it led to another problem as shown below

As you can see, the x-axis is not scaling automatically according to the timestamp range of the data points.

Also, why can't I still see the transformed field "time" values in chrome console? I only see the 2 original elastic index fields as mentioned in my previous comment. How does one see the transformed fields in browser console?

Could you provide the spec?

@aaron-nimocks thanks for your response. Here are the specs. I have reduced the number of data points to 10 so that it can be easily accommodated here. The problem still persists even with 10 data points.

{
  "$schema": "https://vega.github.io/schema/vega/v4.json",
  "description": "A basic scatter plot",
  "padding": 5,
  "data": {
    "name": "source",
    "format": {
      "property": "hits.hits"
    },
    "transform": [
      {
        "type": "formula",
        "as": "time",
        "expr": "toDate(datum._source['@timestamp'])"
      }
    ],
    "url": {
      "index": "alb-logs-*",
      "body": {
        "size": 10,
        "_source": [
          "@timestamp",
          "elb_received_bytes"
        ],
        "query": {
          "bool": {
            "must": [
              {
                "range": {
                  "@timestamp": {
                    "gte": "2021-02-18T00:47:15.704Z",
                    "lte": "2021-02-18T01:02:15.704Z",
                    "format": "strict_date_optional_time"
                  }
                }
              }
            ],
            "filter": [
              {
                "match_all": {}
              }
            ],
            "should": [],
            "must_not": []
          }
        }
      }
    },
    "values": {
      "took": 6,
      "timed_out": false,
      "_shards": {
        "total": 3,
        "successful": 3,
        "skipped": 2,
        "failed": 0
      },
      "hits": {
        "total": 1487,
        "max_score": 1,
        "hits": [
          {
            "_index": "alb-logs-2021.02",
            "_type": "_doc",
            "_id": "6ViesncBHSONY29jDXEY",
            "_score": 1,
            "_source": {
              "@timestamp": "2021-02-18T00:47:36.638Z",
              "elb_received_bytes": "236"
            },
            "time": 1613609256638
          },
          {
            "_index": "alb-logs-2021.02",
            "_type": "_doc",
            "_id": "6liesncBHSONY29jDXEY",
            "_score": 1,
            "_source": {
              "@timestamp": "2021-02-18T00:48:04.753Z",
              "elb_received_bytes": "236"
            },
            "time": 1613609284753
          },
          {
            "_index": "alb-logs-2021.02",
            "_type": "_doc",
            "_id": "61iesncBHSONY29jDXEY",
            "_score": 1,
            "_source": {
              "@timestamp": "2021-02-18T00:47:23.721Z",
              "elb_received_bytes": "202"
            },
            "time": 1613609243721
          },
          {
            "_index": "alb-logs-2021.02",
            "_type": "_doc",
            "_id": "7FiesncBHSONY29jDXEY",
            "_score": 1,
            "_source": {
              "@timestamp": "2021-02-18T00:47:48.237Z",
              "elb_received_bytes": "449"
            },
            "time": 1613609268237
          },
          {
            "_index": "alb-logs-2021.02",
            "_type": "_doc",
            "_id": "7ViesncBHSONY29jDXEY",
            "_score": 1,
            "_source": {
              "@timestamp": "2021-02-18T00:47:53.094Z",
              "elb_received_bytes": "241"
            },
            "time": 1613609273094
          },
          {
            "_index": "alb-logs-2021.02",
            "_type": "_doc",
            "_id": "7liesncBHSONY29jDXEY",
            "_score": 1,
            "_source": {
              "@timestamp": "2021-02-18T00:47:59.482Z",
              "elb_received_bytes": "69"
            },
            "time": 1613609279482
          },
          {
            "_index": "alb-logs-2021.02",
            "_type": "_doc",
            "_id": "71iesncBHSONY29jDXEY",
            "_score": 1,
            "_source": {
              "@timestamp": "2021-02-18T00:48:11.755Z",
              "elb_received_bytes": "165"
            },
            "time": 1613609291755
          },
          {
            "_index": "alb-logs-2021.02",
            "_type": "_doc",
            "_id": "8FiesncBHSONY29jDXEY",
            "_score": 1,
            "_source": {
              "@timestamp": "2021-02-18T00:48:20.981Z",
              "elb_received_bytes": "1373"
            },
            "time": 1613609300981
          },
          {
            "_index": "alb-logs-2021.02",
            "_type": "_doc",
            "_id": "8ViesncBHSONY29jDXEY",
            "_score": 1,
            "_source": {
              "@timestamp": "2021-02-18T00:48:20.258Z",
              "elb_received_bytes": "161"
            },
            "time": 1613609300258
          },
          {
            "_index": "alb-logs-2021.02",
            "_type": "_doc",
            "_id": "8liesncBHSONY29jDXEY",
            "_score": 1,
            "_source": {
              "@timestamp": "2021-02-18T00:48:21.460Z",
              "elb_received_bytes": "90"
            },
            "time": 1613609301460
          }
        ]
      }
    }
  },
  "scales": [
    {
      "name": "x",
      "type": "time",
      "round": true,
      "nice": true,
      "zero": true,
      "domain": {
        "data": "source",
        "field": "time"
      },
      "range": "width"
    },
    {
      "name": "y",
      "type": "linear",
      "round": true,
      "nice": true,
      "zero": true,
      "domain": {
        "data": "source",
        "field": "_source.elb_received_bytes"
      },
      "range": "height"
    }
  ],
  "axes": [
    {
      "scale": "x",
      "grid": true,
      "domain": false,
      "orient": "bottom",
      "tickCount": 5,
      "title": "Time"
    },
    {
      "scale": "y",
      "grid": true,
      "domain": false,
      "orient": "left",
      "titlePadding": 5,
      "title": "Bytes"
    }
  ],
  "marks": [
    {
      "name": "marks",
      "type": "symbol",
      "from": {
        "data": "source"
      },
      "encode": {
        "update": {
          "x": {
            "scale": "x",
            "field": "time"
          },
          "y": {
            "scale": "y",
            "field": "_source.elb_received_bytes"
          },
          "shape": {
            "value": "circle"
          },
          "strokeWidth": {
            "value": 2
          },
          "opacity": {
            "value": 0.5
          },
          "stroke": {
            "value": "#4682b4"
          },
          "fill": {
            "value": "transparent"
          }
        }
      }
    }
  ],
  "config": {
    "range": {
      "category": {
        "scheme": "elastic"
      }
    },
    "arc": {
      "fill": "#54B399"
    },
    "area": {
      "fill": "#54B399"
    },
    "line": {
      "stroke": "#54B399"
    },
    "path": {
      "stroke": "#54B399"
    },
    "rect": {
      "fill": "#54B399"
    },
    "rule": {
      "stroke": "#54B399"
    },
    "shape": {
      "stroke": "#54B399"
    },
    "symbol": {
      "fill": "#54B399"
    },
    "trail": {
      "fill": "#54B399"
    },
    "title": {
      "color": "#343741"
    },
    "style": {
      "guide-label": {
        "fill": "#69707d"
      },
      "guide-title": {
        "fill": "#343741"
      },
      "group-title": {
        "fill": "#343741"
      },
      "group-subtitle": {
        "fill": "#343741"
      }
    },
    "axis": {
      "tickColor": "#eef0f3",
      "domainColor": "#eef0f3",
      "gridColor": "#eef0f3"
    },
    "background": "transparent"
  },
  "width": "container",
  "height": "container",
  "autosize": {
    "type": "fit",
    "contains": "padding"
  }
}

There is a new aggregate called timeunit but it's in V5 and here is the example.

I made all the changes here to make that work. Let me know if this all makes sense.

Got it to work. Thanks for the solution :slight_smile:

1 Like

I am just seeking one more clarification. What is the purpose of following transform?

{
          "type": "aggregate",
          "groupby": ["unit0"],
          "ops": ["average"],
          "fields": ["_source.elb_received_bytes"],
          "as": ["elb_received_bytes"]
        }

When I insert this into my code, I get an error Cannot read property 'elb_received_bytes' of undefined

That one groups the unit0 field that was created with the timeunit aggregation and adds in your aggregation for your y-axis.

So if in timeunit you specify minutes then that aggregation will take the average bytes and group them by unit0 so every minute is accounted for and you can create the chart.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.