Vegalite - Filter Flatten Data

[Siva_Kumar_Menta]

Hi All,

I have data in Elasticsearch index "testdata" with few fields in array format. While trying to apply filter on flattened data @ line #18 as {"filter":"ms > 1"} getting error. Syntax wise it looks fine but not sure about cause of error.

Please suggest modifications required in script.

Error Screenshot:

Sample Index Data:

{"index": {}}
{"SNI#": "HCP1", "filePrefix": "TR37609971", "vehicle": "TE1484625", "ECU": "HCP1_CANFD04", "filedate": "2022-10-18", "threshold": 200, "ECU_SNI": "HCP1_CANFD04-HCP1", "maxms": 69, "minms": 0, "TotalFreq": 0, "avgFreq": 21, "maxFreq": 0, "maxFreqDur": 0, "maxDur": 0, "maxDurFreq": 0, "Frequency": [2, 1, 1, 1], "ms": [0, 14, 20, 69]},{"index": {}},
{"SNI#": "HCP1", "filePrefix": "TR37609901", "vehicle": "TE1484610", "ECU": "HCP1_CANFD04", "filedate": "2022-10-18", "threshold": 200, "ECU_SNI": "HCP1_CANFD04-HCP1", "maxms": 70, "minms": 0, "TotalFreq": 0, "avgFreq": 21, "maxFreq": 0, "maxFreqDur": 0, "maxDur": 0, "maxDurFreq": 0, "Frequency": [2, 1, 1], "ms": [0, 14, 70]}

Syntax:

{
  "$schema": "https://vega.github.io/schema/vega-lite/v5.json",
  "title": "",
  "data": {
    "url": {
      "index": "testdata",
      "%context%": true,  "body": {"size": 10000}
    },
    "format": {"property": "hits.hits"}
  }, 
   "transform": [
    {"calculate": "datum._source.ECU_SNI", "as": "ECU_SNI"},
    {"calculate": "datum._source.ECU", "as": "ECU"},    
    {"calculate": "datum._source.ms", "as": "ms"},
    {"calculate": "datum._source.Frequency", "as": "Frequency"},
    {"flatten": ["ms", "Frequency"] },

    {"filter":"ms > 1"},
    {
      "aggregate": [{
       "op": "sum",
       "field": "Frequency",
       "as": "TotFrequency"
      }],
      "groupby": ["ECU_SNI","ECU"] }
      ],
  
  "layer": [{
   "mark": {"type": "bar", "filled": true},
  "encoding": {
    
    "x": {"field": "TotFrequency", "type": "quantitative","title":"Frequency","aggregate": "sum"},
    "y": {"field": "ECU_SNI", "type": "nominal","title": "ECU_SNI"},
    
    "color": {"field": "ECU","title":"ECU","orient":"top"},
    
    "tooltip": [
      {"field": "ECU_SNI", "type": "ordinal", "title": "SNI", "tooltip": true},
      {"field": "TotFrequency", "type": "quantitative", "title": "frequency", "tooltip": true}
    ]
  }
 }],
  "config": {
    "legend": {"orient": "top", "layout": {"top": {"anchor": "left"}}}
  }
}

Thanks for reading through my problem / your help!
Siva Kumar

[Siva_Kumar_Menta]

@Siva_Kumar_Menta

{"filter":"ms > 1"},

Hi All,

Any inputs on the problem posted. Please note that I have a deliverable in 2 days so please suggest whether it's possible to apply filter on flattened data. If not any alternatives.

Looking forward for responses.

Thanks for your help!
Siva Kumar