Transform / Filter works with inline data - Does not work with identical index data

I am working with two versions of a Vega Detail/Overview (Area Chart) Visualization.
They both have the same dropdowns that are used to filter the data:

"signals": [
    { "name": "detailDomain" },
    { "name": "regionFilter", "value": "EMEA",
      "bind": {"input": "select", "options": ["AMER", "EMEA", "APAC"]} },
    { "name": "acctypeFilter", "value": "StrategyAccount",
      "bind": {"input": "select", "options": ["StrategyAccount", "Strategy", "Account"]} }
  ],

The data source of the version where filtering works:

  "data": [
    {
      "name": "sp500",
      "values": [
	{"account":"XXXXXX","price":-1.387814312978290e+004,"acctype":"Account","region":"APAC"},
	{"account":"YYYYY","price":0.000000000000000e+000,"acctype":"Account","region":"APAC"},
	{"account":"ZZZZZ","price":-6.588935784852400e+005,"acctype":"Account","region":"APAC"},
	{"account":"X1X1X1","price":8.863933226918270e+005,"acctype":"Account","region":"APAC"},
	{"account":"Z2Z2Z2","price":-3.314336740809040e+005,"acctype":"Account","region":"APAC"},
        # . etc, etc, 
	#
    ]
    },

The data source of the version where filtering does not work:

  "data": [
    {
      "name": "sp500",
	    url: {
        %context%: true
        %timefield%: @timestamp
        index: mgpnlquery*
        body: {
          size: 1000 
          _source: ["@timestamp", "price", "account", "acctype", "region" ]
      }
    }
	 "format": { "property": "hits.hits" }
    },
    # Here are the data filters:
    {
      "name": "byAccType",
      "source": "sp500",
      "transform": [
        {"type": "filter", "expr": "datum.acctype == acctypeFilter"}
      ]
    },
    {
      "name": "byFilter",
      "source": "byAccType",
      "transform": [
        {"type": "filter", "expr": "datum.region == regionFilter"}
      ]
    }    
  ],

The "byFilter" data set works in the inline data version, but results in blank data in the index data version shown below: (Note - substituting "sp500" for "byFilter" will make the (unfiltered) display work). I am suspecting that the Transform / Filter behaves differently for inline data vs. index data.)

  "marks": [
    {
      "type": "group",
      "name": "detail",
      "encode": {
        "enter": {
          "height": {"value": 390},
          "width": {"value": 1700} 
        }
      },
      "scales": [
        {
          "name": "xDetail",
          "type": "point",
          "range": "width",
          "domain": {"data": "byFilter", "field": "_source.account"}
          "domainRaw": {"signal": "detailDomain"}
        },
        {
          "name": "yDetail",
          "type": "linear",
          "range": [390, 0],
          "domain": {"data": "byFilter", "field": "_source.price"},
          "nice": true, "zero": true
        }
      ],
      "axes": [
        {"orient": "bottom", "scale": "xDetail", "grid": "true", "labelOverlap": true,
          "title": "Accounts"},
        {"orient": "left", "scale": "yDetail", "grid": "true",
          "title": "PnL"}
      ],
      "marks": [
        {
          "type": "group",
          "encode": {
            "enter": {
              "height": {"field": {"group": "height"}},
              "width": {"field": {"group": "width"}},
              "clip": {"value": true}
            }
          },
          "marks": [
            {
              "type": "area",
              "from": {"data": "byFilter"},
              "encode": {
                "update": {
                  "x": {"scale": "xDetail", "field": "_source.account"},
                  "y": {"scale": "yDetail", "field": "_source.price"},
                  "y2": {"scale": "yDetail", "value": 0},
                  "fill": {"value": "steelblue"}
                }
              }
            }
          ]
        }
      ]
    },

This is what the index data looks like:

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 23870,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "mgpnlquery-2019.04.03",
        "_type" : "doc",
        "_id" : "IyX84mkBLZb84o2j7mwW",
        "_score" : 1.0,
        "_source" : {
          "region" : "APAC",
          "price" : -2071.39358048752,
          "@version" : "1",
          "tags" : [
            "PnlQuery2"
          ],
          "@timestamp" : "2019-04-03T11:37:00.592Z",
          "type" : "PnlQuery2",
          "account" : "ZZZZZ",
          "acctype" : "StrategyAccount"
        }
      },
      {
        "_index" : "mgpnlquery-2019.04.03",
        "_type" : "doc",
        "_id" : "JSX84mkBLZb84o2j7mwX",
        "_score" : 1.0,
        "_source" : {
          "region" : "APAC",
          "price" : 5640.71772310075,
          "@version" : "1",
          "tags" : [
            "PnlQuery2"
          ],
          "@timestamp" : "2019-04-03T11:37:00.591Z",
          "type" : "PnlQuery2",
          "account" : "ZZZZ",
          "acctype" : "StrategyAccount"
        }
      },
      {
        "_index" : "mgpnlquery-2019.04.03",

Michael, it wouldn't be possible for inline vs query to behave differently because internally Kibana replaces the url with values before even sending it to Vega. So unless the data in values is different from what url returns, it should behave identically. Take a look at https://gist.github.com/nyurik/736c34970ba3c32f3fe3d45d66719b3e -- check if the data shown in the debug console is what you expect.

Yuri,
Data sample next email.
It works when no filter is applied. When filter changed to "sp500" to "byFilter" in the scales and area section in Marks, no data is displayed.

{
"$schema": "https://vega.github.i/schema/vega/v3.json",
"width": 1700,
"height": 480,
"padding": 5,
"autosize": "pad",

"data": [
{
"name": "sp500",
url: {
index: mgpnlquery*
body: {
size: 1000
_source: ["price", "account", "acctype", "region" ]
"query": {
"range" : {
"@timestamp" : {
"gte" : "now-34s",
"lt" : "now"
}
}
}
}
}
"format": { "property": "hits.hits" }
},
{
"name": "byAccType",
"source": "sp500",
"transform": [
{"type": "filter", "expr": "datum.acctype == acctypeFilter"}
]
},
{
"name": "byFilter",
"source": "byAccType",
"transform": [
{"type": "filter", "expr": "datum.region == regionFilter"}
]
}
],

"title": {
"text": {"signal": "'Accounts vs Pnl - region: ' + regionFilter + ' / account Type: ' + acctypeFilter"}
},

"signals": [
{ "name": "detailDomain" },
{ "name": "regionFilter", "value": "EMEA",
"bind": {"input": "select", "options": ["AMER", "EMEA", "APAC"]} },
{ "name": "acctypeFilter", "value": "StrategyAccount",
"bind": {"input": "select", "options": ["StrategyAccount", "Strategy", "Account"]} }
],

"marks": [
{
"type": "group",
"name": "detail",
"encode": {
"enter": {
"height": {"value": 390},
"width": {"value": 1700}
}
},
"scales": [
{
"name": "xDetail",
"type": "point",
"range": "width",
"domain": {"data": "sp500", "field": "_source.account"}
"domainRaw": {"signal": "detailDomain"}
},
{
"name": "yDetail",
"type": "linear",
"range": [390, 0],
"domain": {"data": "sp500", "field": "_source.price"},
"nice": true, "zero": true
}
],
"axes": [
{"orient": "bottom", "scale": "xDetail", "grid": "true", "labelOverlap": true,
"title": "Accounts"},
{"orient": "left", "scale": "yDetail", "grid": "true",
"title": "PnL"}
],
"marks": [
{
"type": "group",
"encode": {
"enter": {
"height": {"field": {"group": "height"}},
"width": {"field": {"group": "width"}},
"clip": {"value": true}
}
},
"marks": [
{
"type": "area",
"from": {"data": "sp500"},
"encode": {
"update": {
"x": {"scale": "xDetail", "field": "_source.account"},
"y": {"scale": "yDetail", "field": "_source.price"},
"y2": {"scale": "yDetail", "value": 0},
"fill": {"value": "steelblue"}
}
}
}
]
}
]
},

{
  "type": "group",
  "name": "overview",
  "encode": {
    "enter": {
      "x": {"value": 0},
      "y": {"value": 430},
      "height": {"value": 70},
      "width": {"value": 1700},
      "fill": {"value": "transparent"}
    }
  },
  "signals": [
    {
      "name": "brush", "value": 0,
      "on": [
        {
          "events": "@overview:mousedown",
          "update": "[x(), x()]"
        },
        {
          "events": "[@overview:mousedown, window:mouseup] > window:mousemove!",
          "update": "[brush[0], clamp(x(), 0, width)]"
        },
        {
          "events": {"signal": "delta"},
          "update": "clampRange([anchor[0] + delta, anchor[1] + delta], 0, width)"
        }
      ]
    },
    {
      "name": "anchor", "value": null,
      "on": [{"events": "@brush:mousedown", "update": "slice(brush)"}]
    },
    {
      "name": "xdown", "value": 0,
      "on": [{"events": "@brush:mousedown", "update": "x()"}]
    },
    {
      "name": "delta", "value": 0,
      "on": [
        {
          "events": "[@brush:mousedown, window:mouseup] > window:mousemove!",
          "update": "x() - xdown"
        }
      ]
    },
    {
      "name": "detailDomain",
      "push": "outer",
      "on": [
        {
          "events": {"signal": "brush"},
          "update": "span(brush) ? invert('xOverview', brush) : null"
        }
      ]
    }
  ],
  "scales": [
    {
      "name": "xOverview",
      "type": "point",
      "range": "width",
      "domain": {"data": "sp500", "field": "_source.account"}
    },
    {
      "name": "yOverview",
      "type": "linear",
      "range": [70, 0],
      "domain": {"data": "sp500", "field": "_source.price"},
      "nice": true, "zero": true
    }
  ],
  "axes": [
    {"orient": "bottom", "scale": "xOverview", "labelOverlap": true}
  ],
  "marks": [
    {
      "type": "area",
      "interactive": false,
      "from": {"data": "sp500"},
      "encode": {
        "update": {
          "x": {"scale": "xOverview", "field": "_source.account"},
          "y": {"scale": "yOverview", "field": "_source.price"},
          "y2": {"scale": "yOverview", "value": 0},
          "fill": {"value": "firebrick"}
        }
      }
    },
    {
      "type": "rect",
      "name": "brush",
      "encode": {
        "enter": {
          "y": {"value": 0},
          "height": {"value": 70},
          "fill": {"value": "#333"},
          "fillOpacity": {"value": 0.2}
        },
        "update": {
          "x": {"signal": "brush[0]"},
          "x2": {"signal": "brush[1]"}
        }
      }
    },
    {
      "type": "rect",
      "interactive": false,
      "encode": {
        "enter": {
          "y": {"value": 0},
          "height": {"value": 70},
          "width": {"value": 1},
          "fill": {"value": "firebrick"}
        },
        "update": {
          "x": {"signal": "brush[0]"}
        }
      }
    },
    {
      "type": "rect",
      "interactive": false,
      "encode": {
        "enter": {
          "y": {"value": 0},
          "height": {"value": 70},
          "width": {"value": 1},
          "fill": {"value": "firebrick"}
        },
        "update": {
          "x": {"signal": "brush[1]"}
        }
      }
    }
  ]
}

]
}

Data sample for above code example:
{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 91190,
"max_score" : 1.0,
"hits" : [
{
"_index" : "mgpnlquery-2019.04.03",
"_type" : "doc",
"_id" : "IyX84mkBLZb84o2j7mwW",
"_score" : 1.0,
"_source" : {
"region" : "APAC",
"price" : -2071.39358048752,
"@version" : "1",
"tags" : [
"PnlQuery2"
],
"@timestamp" : "2019-04-03T11:37:00.592Z",
"type" : "PnlQuery2",
"account" : "YZM1",
"acctype" : "StrategyAccount"
}
},
{
"_index" : "mgpnlquery-2019.04.03",
"_type" : "doc",
"_id" : "JSX84mkBLZb84o2j7mwX",
"_score" : 1.0,
"_source" : {
"region" : "APAC",
"price" : 5640.71772310075,
"@version" : "1",
"tags" : [
"PnlQuery2"
],
"@timestamp" : "2019-04-03T11:37:00.591Z",
"type" : "PnlQuery2",
"account" : "IAB3",
"acctype" : "StrategyAccount"
}
},
{
"_index" : "mgpnlquery-2019.04.03",
"_type" : "doc",
"_id" : "JyX84mkBLZb84o2j7mwX",
"_score" : 1.0,
"_source" : {
"region" : "APAC",
"price" : 34380.5662564032,
"@version" : "1",
"tags" : [
"PnlQuery2"
],
"@timestamp" : "2019-04-03T11:37:00.591Z",
"type" : "PnlQuery2",
"account" : "REB11",
"acctype" : "StrategyAccount"
}
},
{
"_index" : "mgpnlquery-2019.04.03",
"_type" : "doc",
"_id" : "LSX84mkBLZb84o2j7mwX",
"_score" : 1.0,
"_source" : {
"region" : "APAC",
"price" : 326.803996763185,
"@version" : "1",
"tags" : [
"PnlQuery2"
],
"@timestamp" : "2019-04-03T11:37:00.592Z",
"type" : "PnlQuery2",
"account" : "WBSM20",
"acctype" : "StrategyAccount"
}
},
{
"_index" : "mgpnlquery-2019.04.03",
"_type" : "doc",
"_id" : "MCX84mkBLZb84o2j7mwY",
"_score" : 1.0,
"_source" : {
"region" : "APAC",
"price" : 122.08204961894,
"@version" : "1",
"tags" : [
"PnlQuery2"
],
"@timestamp" : "2019-04-03T11:37:00.590Z",
"type" : "PnlQuery2",
"account" : "FCT3",
"acctype" : "StrategyAccount"
}
},
{
"_index" : "mgpnlquery-2019.04.03",
"_type" : "doc",
"_id" : "MyX84mkBLZb84o2j7mwY",
"_score" : 1.0,
"_source" : {
"region" : "APAC",
"price" : 936.341516233648,
"@version" : "1",
"tags" : [
"PnlQuery2"
],
"@timestamp" : "2019-04-03T11:37:00.591Z",
"type" : "PnlQuery2",
"account" : "IABM7",
"acctype" : "StrategyAccount"
}
},
{
"_index" : "mgpnlquery-2019.04.03",
"_type" : "doc",
"_id" : "OCX84mkBLZb84o2j7mwY",
"_score" : 1.0,
"_source" : {
"region" : "APAC",
"price" : -431.228101697961,
"@version" : "1",
"tags" : [
"PnlQuery2"
],
"@timestamp" : "2019-04-03T11:37:00.591Z",
"type" : "PnlQuery2",
"account" : "REB9",
"acctype" : "StrategyAccount"
}
},
{
"_index" : "mgpnlquery-2019.04.03",
"_type" : "doc",
"_id" : "SCX84mkBLZb84o2j7mwa",
"_score" : 1.0,
"_source" : {
"region" : "APAC",
"price" : 108285.374512865,
"@version" : "1",
"tags" : [
"PnlQuery2"
],
"@timestamp" : "2019-04-03T11:37:00.590Z",
"type" : "PnlQuery2",
"account" : "GRPM2AS",
"acctype" : "Account"
}
},
{
"_index" : "mgpnlquery-2019.04.03",
"_type" : "doc",
"_id" : "UyX84mkBLZb84o2j7mwa",
"_score" : 1.0,
"_source" : {
"region" : "APAC",
"price" : -41.3260264127212,
"@version" : "1",
"tags" : [
"PnlQuery2"
],
"@timestamp" : "2019-04-03T11:37:00.591Z",
"type" : "PnlQuery2",
"account" : "REBM14",
"acctype" : "StrategyAccount"
}
},
{
"_index" : "mgpnlquery-2019.04.03",
"_type" : "doc",
"_id" : "ViX84mkBLZb84o2j7mwa",
"_score" : 1.0,
"_source" : {
"region" : "APAC",
"price" : -2859.80967665044,
"@version" : "1",
"tags" : [
"PnlQuery2"
],
"@timestamp" : "2019-04-03T11:37:00.591Z",
"type" : "PnlQuery2",
"account" : "REBM4",
"acctype" : "StrategyAccount"
}
}
]
}
}

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.