[BUG?] Escaping periods in vega-lite calculate transform

(Tadgh) #1

Using the percentile aggregate, elasticsearch returns results wherein the key contains a period. Like so:

 "score_outlier": {
            "values": {
              "1.0": 1,
              "5.0": 1,
              "25.0": 2,
              "50.0": 3,
              "75.0": 3,
              "95.0": 5,
              "99.0": 5

When visualizing, vega lite indicates that you should escape any fields containing periods using a double backslash: \\.

This works totally fine in the encodings, like here:

      "mark": {"type": "bar", "style": "box"},
      "encoding": {
        "y": {"field": "score_outlier.values.25\\.0", "type": "quantitative"},
        "y2": {"field": "score_outlier.values.75\\.0", "type": "quantitative"},
        "x": {"field": "key", "type": "nominal"},
        "color": {"value": "steelblue"},
        "size": {"value": 11}

But if I do the same thing during a calculate transform, I get a Parse Error from Vega. Here is an example which fails:

   "transform": [
       "calculate": "datum.score_outlier.values.75\\.0 - datum.score_outlier.values.25\\.0",
       "as": "iqr"

Which returns:

Expression parse error: "datum.score_outlier.values.75\\.0 - datum.score_outlier.values.25\\.0"

I've tried not escaping, double-escaping, using square bracket indexing, but nothing seems to work. Am I missing something? The moment I switch this calculation to use a field that doesn't contain a period, all is well.

Thanks for any insights

(kulkarni) #2

@nyuriks - any thoughts here?

(Tadgh) #3

To quick-fix the issue, I tried just adding a bucket script aggregation, which turns out, suffers the same issue I am seeing here I think.

I actually think the issue may be higher up, in Elasticsearch: Bucket Script Aggregating on fields that contain a period . To narrow this down, I made a small example query. This may not be VEGA-related after all, but just ES related.

(Yuri Astrakhan) #4

@tadgh for the Vega part, try using index access instead, e.g. datum.score_outlier.values['75.0']

(Tadgh) #5

Thanks, changing my transform to:

  "transform": [
      "calculate": "datum.score_outlier.values['75.0'] - datum.score_outlier.values['25.0']"
      "as": "iqr"

Did the trick. That said, seems a bit off that its fine in the encoding, but breaks in the calculation. Slightly unintuitive to have to nest quotes in one spot, but able to escape in another.

(Yuri Astrakhan) #6

@tadgh while not ideal, the reason it works in the field is because field is not an expression, it is a path to a value inside the current datum. So if datum is {a: {b: [10.5]: 'something'}}, getting to something would be a.b.10\\.5. On the other hand, signal and expr values are real expressions - they get evaluated similar to how javascript evaluates things. JS would throw an error with datum.a.b.10\\.5, and so does Vega.

(Tadgh) #7

Understood, thanks for the clarification

(system) #8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.