# Transform / Filter works with inline data - Does not work with identical index data **URL:** \ **Category:** Kibana\ **Tags:** vega\ **Created:** [April 2, 2019, 2:28pm UTC](https://discuss.elastic.co/t/transform-filter-works-with-inline-data-does-not-work-with-identical-index-data/175021 "2019-04-02T14:28:21Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![mgolubov55](https://avatars.discourse-cdn.com/v4/letter/m/c6cbf5/32.png) [@mgolubov55](https://discuss.elastic.co/u/mgolubov55)\ **Post date:** [April 2, 2019, 2:28pm UTC](https://discuss.elastic.co/t/transform-filter-works-with-inline-data-does-not-work-with-identical-index-data/175021/1 "2019-04-02T14:28:21Z")
I am working with two versions of a Vega Detail/Overview (Area Chart) Visualization. They both have the same dropdowns that are used to filter the data: ``` "signals": [ { "name": "detailDomain" }, { "name": "regionFilter", "value": "EMEA", "bind": {"input": "select", "options": ["AMER", "EMEA", "APAC"]} }, { "name": "acctypeFilter", "value": "StrategyAccount", "bind": {"input": "select", "options": ["StrategyAccount", "Strategy", "Account"]} } ], ``` The data source of the version where filtering works: ``` "data": [ { "name": "sp500", "values": [ {"account":"XXXXXX","price":-1.387814312978290e+004,"acctype":"Account","region":"APAC"}, {"account":"YYYYY","price":0.000000000000000e+000,"acctype":"Account","region":"APAC"}, {"account":"ZZZZZ","price":-6.588935784852400e+005,"acctype":"Account","region":"APAC"}, {"account":"X1X1X1","price":8.863933226918270e+005,"acctype":"Account","region":"APAC"}, {"account":"Z2Z2Z2","price":-3.314336740809040e+005,"acctype":"Account","region":"APAC"}, # . etc, etc, # ] }, ``` The data source of the version where filtering does not work: ``` "data": [ { "name": "sp500", url: { %context%: true %timefield%: @timestamp index: mgpnlquery* body: { size: 1000 _source: ["@timestamp", "price", "account", "acctype", "region"] } } "format": { "property": "hits.hits" } }, # Here are the data filters: { "name": "byAccType", "source": "sp500", "transform": [ {"type": "filter", "expr": "datum.acctype == acctypeFilter"} ] }, { "name": "byFilter", "source": "byAccType", "transform": [ {"type": "filter", "expr": "datum.region == regionFilter"} ] } ], ``` The "byFilter" data set works in the inline data version, but results in blank data in the index data version shown below: (Note - substituting "sp500" for "byFilter" will make the (unfiltered) display work). I am suspecting that the Transform / Filter behaves differently for inline data vs. index data.) ``` "marks": [ { "type": "group", "name": "detail", "encode": { "enter": { "height": {"value": 390}, "width": {"value": 1700} } }, "scales": [ { "name": "xDetail", "type": "point", "range": "width", "domain": {"data": "byFilter", "field": "_source.account"} "domainRaw": {"signal": "detailDomain"} }, { "name": "yDetail", "type": "linear", "range": [390, 0], "domain": {"data": "byFilter", "field": "_source.price"}, "nice": true, "zero": true } ], "axes": [ {"orient": "bottom", "scale": "xDetail", "grid": "true", "labelOverlap": true, "title": "Accounts"}, {"orient": "left", "scale": "yDetail", "grid": "true", "title": "PnL"} ], "marks": [ { "type": "group", "encode": { "enter": { "height": {"field": {"group": "height"}}, "width": {"field": {"group": "width"}}, "clip": {"value": true} } }, "marks": [ { "type": "area", "from": {"data": "byFilter"}, "encode": { "update": { "x": {"scale": "xDetail", "field": "_source.account"}, "y": {"scale": "yDetail", "field": "_source.price"}, "y2": {"scale": "yDetail", "value": 0}, "fill": {"value": "steelblue"} } } } ] } ] }, ``` This is what the index data looks like: ``` { "took" : 0, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 23870, "max_score" : 1.0, "hits" : [ { "_index" : "mgpnlquery-2019.04.03", "_type" : "doc", "_id" : "IyX84mkBLZb84o2j7mwW", "_score" : 1.0, "_source" : { "region" : "APAC", "price" : -2071.39358048752, "@version" : "1", "tags" : [ "PnlQuery2" ], "@timestamp" : "2019-04-03T11:37:00.592Z", "type" : "PnlQuery2", "account" : "ZZZZZ", "acctype" : "StrategyAccount" } }, { "_index" : "mgpnlquery-2019.04.03", "_type" : "doc", "_id" : "JSX84mkBLZb84o2j7mwX", "_score" : 1.0, "_source" : { "region" : "APAC", "price" : 5640.71772310075, "@version" : "1", "tags" : [ "PnlQuery2" ], "@timestamp" : "2019-04-03T11:37:00.591Z", "type" : "PnlQuery2", "account" : "ZZZZ", "acctype" : "StrategyAccount" } }, { "_index" : "mgpnlquery-2019.04.03", ``` ---
**Author:** ![nyuriks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nyuriks/32/26171_2.png) [@nyuriks](https://discuss.elastic.co/u/nyuriks)\ **Post date:** [April 3, 2019, 4:10pm UTC](https://discuss.elastic.co/t/transform-filter-works-with-inline-data-does-not-work-with-identical-index-data/175021/2 "2019-04-03T16:10:17Z")
Michael, it wouldn't be possible for inline vs query to behave differently because internally Kibana replaces the `url` with `values` before even sending it to Vega. So unless the data in `values` is different from what `url` returns, it should behave identically. Take a look at [https://gist.github.com/nyurik/736c34970ba3c32f3fe3d45d66719b3e](https://gist.github.com/nyurik/736c34970ba3c32f3fe3d45d66719b3e) -- check if the data shown in the debug console is what you expect. ---
**Author:** ![mgolubov55](https://avatars.discourse-cdn.com/v4/letter/m/c6cbf5/32.png) [@mgolubov55](https://discuss.elastic.co/u/mgolubov55)\ **Post date:** [April 3, 2019, 5:49pm UTC](https://discuss.elastic.co/t/transform-filter-works-with-inline-data-does-not-work-with-identical-index-data/175021/3 "2019-04-03T17:49:43Z")
Yuri, Data sample next email. It works when no filter is applied. When filter changed to "sp500" to "byFilter" in the scales and area section in Marks, no data is displayed. { "$schema": "[https://vega.github.i/schema/vega/v3.json](https://vega.github.i/schema/vega/v3.json)", "width": 1700, "height": 480, "padding": 5, "autosize": "pad", "data": [ { "name": "sp500", url: { index: mgpnlquery\* body: { size: 1000 \_source: ["price", "account", "acctype", "region"] "query": { "range" : { "@timestamp" : { "gte" : "now-34s", "lt" : "now" } } } } } "format": { "property": "hits.hits" } }, { "name": "byAccType", "source": "sp500", "transform": [ {"type": "filter", "expr": "datum.acctype == acctypeFilter"} ] }, { "name": "byFilter", "source": "byAccType", "transform": [ {"type": "filter", "expr": "datum.region == regionFilter"} ] } ], "title": { "text": {"signal": "'Accounts vs Pnl - region: ' + regionFilter + ' / account Type: ' + acctypeFilter"} }, "signals": [ { "name": "detailDomain" }, { "name": "regionFilter", "value": "EMEA", "bind": {"input": "select", "options": ["AMER", "EMEA", "APAC"]} }, { "name": "acctypeFilter", "value": "StrategyAccount", "bind": {"input": "select", "options": ["StrategyAccount", "Strategy", "Account"]} } ], "marks": [ { "type": "group", "name": "detail", "encode": { "enter": { "height": {"value": 390}, "width": {"value": 1700} } }, "scales": [ { "name": "xDetail", "type": "point", "range": "width", "domain": {"data": "sp500", "field": "\_source.account"} "domainRaw": {"signal": "detailDomain"} }, { "name": "yDetail", "type": "linear", "range": [390, 0], "domain": {"data": "sp500", "field": "\_source.price"}, "nice": true, "zero": true } ], "axes": [ {"orient": "bottom", "scale": "xDetail", "grid": "true", "labelOverlap": true, "title": "Accounts"}, {"orient": "left", "scale": "yDetail", "grid": "true", "title": "PnL"} ], "marks": [ { "type": "group", "encode": { "enter": { "height": {"field": {"group": "height"}}, "width": {"field": {"group": "width"}}, "clip": {"value": true} } }, "marks": [ { "type": "area", "from": {"data": "sp500"}, "encode": { "update": { "x": {"scale": "xDetail", "field": "\_source.account"}, "y": {"scale": "yDetail", "field": "\_source.price"}, "y2": {"scale": "yDetail", "value": 0}, "fill": {"value": "steelblue"} } } } ] } ] }, ``` { "type": "group", "name": "overview", "encode": { "enter": { "x": {"value": 0}, "y": {"value": 430}, "height": {"value": 70}, "width": {"value": 1700}, "fill": {"value": "transparent"} } }, "signals": [ { "name": "brush", "value": 0, "on": [ { "events": "@overview:mousedown", "update": "[x(), x()]" }, { "events": "[@overview:mousedown, window:mouseup] > window:mousemove!", "update": "[brush[0], clamp(x(), 0, width)]" }, { "events": {"signal": "delta"}, "update": "clampRange([anchor[0] + delta, anchor[1] + delta], 0, width)" } ] }, { "name": "anchor", "value": null, "on": [{"events": "@brush:mousedown", "update": "slice(brush)"}] }, { "name": "xdown", "value": 0, "on": [{"events": "@brush:mousedown", "update": "x()"}] }, { "name": "delta", "value": 0, "on": [ { "events": "[@brush:mousedown, window:mouseup] > window:mousemove!", "update": "x() - xdown" } ] }, { "name": "detailDomain", "push": "outer", "on": [ { "events": {"signal": "brush"}, "update": "span(brush) ? invert('xOverview', brush) : null" } ] } ], "scales": [ { "name": "xOverview", "type": "point", "range": "width", "domain": {"data": "sp500", "field": "_source.account"} }, { "name": "yOverview", "type": "linear", "range": [70, 0], "domain": {"data": "sp500", "field": "_source.price"}, "nice": true, "zero": true } ], "axes": [ {"orient": "bottom", "scale": "xOverview", "labelOverlap": true} ], "marks": [ { "type": "area", "interactive": false, "from": {"data": "sp500"}, "encode": { "update": { "x": {"scale": "xOverview", "field": "_source.account"}, "y": {"scale": "yOverview", "field": "_source.price"}, "y2": {"scale": "yOverview", "value": 0}, "fill": {"value": "firebrick"} } } }, { "type": "rect", "name": "brush", "encode": { "enter": { "y": {"value": 0}, "height": {"value": 70}, "fill": {"value": "#333"}, "fillOpacity": {"value": 0.2} }, "update": { "x": {"signal": "brush[0]"}, "x2": {"signal": "brush[1]"} } } }, { "type": "rect", "interactive": false, "encode": { "enter": { "y": {"value": 0}, "height": {"value": 70}, "width": {"value": 1}, "fill": {"value": "firebrick"} }, "update": { "x": {"signal": "brush[0]"} } } }, { "type": "rect", "interactive": false, "encode": { "enter": { "y": {"value": 0}, "height": {"value": 70}, "width": {"value": 1}, "fill": {"value": "firebrick"} }, "update": { "x": {"signal": "brush[1]"} } } } ] } ``` ] } ---
**Author:** ![mgolubov55](https://avatars.discourse-cdn.com/v4/letter/m/c6cbf5/32.png) [@mgolubov55](https://discuss.elastic.co/u/mgolubov55)\ **Post date:** [April 3, 2019, 6:02pm UTC](https://discuss.elastic.co/t/transform-filter-works-with-inline-data-does-not-work-with-identical-index-data/175021/4 "2019-04-03T18:02:41Z")
Data sample for above code example: { "took" : 0, "timed\_out" : false, "\_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 91190, "max\_score" : 1.0, "hits" : [ { "\_index" : "mgpnlquery-2019.04.03", "\_type" : "doc", "\_id" : "IyX84mkBLZb84o2j7mwW", "\_score" : 1.0, "\_source" : { "region" : "APAC", "price" : -2071.39358048752, "@version" : "1", "tags" : [ "PnlQuery2" ], "@timestamp" : "2019-04-03T11:37:00.592Z", "type" : "PnlQuery2", "account" : "YZM1", "acctype" : "StrategyAccount" } }, { "\_index" : "mgpnlquery-2019.04.03", "\_type" : "doc", "\_id" : "JSX84mkBLZb84o2j7mwX", "\_score" : 1.0, "\_source" : { "region" : "APAC", "price" : 5640.71772310075, "@version" : "1", "tags" : [ "PnlQuery2" ], "@timestamp" : "2019-04-03T11:37:00.591Z", "type" : "PnlQuery2", "account" : "IAB3", "acctype" : "StrategyAccount" } }, { "\_index" : "mgpnlquery-2019.04.03", "\_type" : "doc", "\_id" : "JyX84mkBLZb84o2j7mwX", "\_score" : 1.0, "\_source" : { "region" : "APAC", "price" : 34380.5662564032, "@version" : "1", "tags" : [ "PnlQuery2" ], "@timestamp" : "2019-04-03T11:37:00.591Z", "type" : "PnlQuery2", "account" : "REB11", "acctype" : "StrategyAccount" } }, { "\_index" : "mgpnlquery-2019.04.03", "\_type" : "doc", "\_id" : "LSX84mkBLZb84o2j7mwX", "\_score" : 1.0, "\_source" : { "region" : "APAC", "price" : 326.803996763185, "@version" : "1", "tags" : [ "PnlQuery2" ], "@timestamp" : "2019-04-03T11:37:00.592Z", "type" : "PnlQuery2", "account" : "WBSM20", "acctype" : "StrategyAccount" } }, { "\_index" : "mgpnlquery-2019.04.03", "\_type" : "doc", "\_id" : "MCX84mkBLZb84o2j7mwY", "\_score" : 1.0, "\_source" : { "region" : "APAC", "price" : 122.08204961894, "@version" : "1", "tags" : [ "PnlQuery2" ], "@timestamp" : "2019-04-03T11:37:00.590Z", "type" : "PnlQuery2", "account" : "FCT3", "acctype" : "StrategyAccount" } }, { "\_index" : "mgpnlquery-2019.04.03", "\_type" : "doc", "\_id" : "MyX84mkBLZb84o2j7mwY", "\_score" : 1.0, "\_source" : { "region" : "APAC", "price" : 936.341516233648, "@version" : "1", "tags" : [ "PnlQuery2" ], "@timestamp" : "2019-04-03T11:37:00.591Z", "type" : "PnlQuery2", "account" : "IABM7", "acctype" : "StrategyAccount" } }, { "\_index" : "mgpnlquery-2019.04.03", "\_type" : "doc", "\_id" : "OCX84mkBLZb84o2j7mwY", "\_score" : 1.0, "\_source" : { "region" : "APAC", "price" : -431.228101697961, "@version" : "1", "tags" : [ "PnlQuery2" ], "@timestamp" : "2019-04-03T11:37:00.591Z", "type" : "PnlQuery2", "account" : "REB9", "acctype" : "StrategyAccount" } }, { "\_index" : "mgpnlquery-2019.04.03", "\_type" : "doc", "\_id" : "SCX84mkBLZb84o2j7mwa", "\_score" : 1.0, "\_source" : { "region" : "APAC", "price" : 108285.374512865, "@version" : "1", "tags" : [ "PnlQuery2" ], "@timestamp" : "2019-04-03T11:37:00.590Z", "type" : "PnlQuery2", "account" : "GRPM2AS", "acctype" : "Account" } }, { "\_index" : "mgpnlquery-2019.04.03", "\_type" : "doc", "\_id" : "UyX84mkBLZb84o2j7mwa", "\_score" : 1.0, "\_source" : { "region" : "APAC", "price" : -41.3260264127212, "@version" : "1", "tags" : [ "PnlQuery2" ], "@timestamp" : "2019-04-03T11:37:00.591Z", "type" : "PnlQuery2", "account" : "REBM14", "acctype" : "StrategyAccount" } }, { "\_index" : "mgpnlquery-2019.04.03", "\_type" : "doc", "\_id" : "ViX84mkBLZb84o2j7mwa", "\_score" : 1.0, "\_source" : { "region" : "APAC", "price" : -2859.80967665044, "@version" : "1", "tags" : [ "PnlQuery2" ], "@timestamp" : "2019-04-03T11:37:00.591Z", "type" : "PnlQuery2", "account" : "REBM4", "acctype" : "StrategyAccount" } } ] } } ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 1, 2019, 6:02pm UTC](https://discuss.elastic.co/t/transform-filter-works-with-inline-data-does-not-work-with-identical-index-data/175021/5 "2019-05-01T18:02:53Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.