Vega Is Not Showing Anything Despite Code Passing Error-Free

im unable to get my Vega to show smth?


this the o/p using Vega-Lite 2.0


this is the o/p using Vega 4.0

{
"$schema": "https://vega.github.io/schema/vega/v4.json",
"data": {
  "url": {
    "%context%": true,
    "%timefield%": "DateTime",
    "index": "nsew*",
    "body": {
      "size": 10000,
      "_source": ["Current_Platform_Name", "Align_Distance_To_Go_mm","Train_Type"]
    }
  },
  "format": {"property": "hits.hits"}
},
"transform": [
 {"filter": {"field": "datum._source['Align_Distance_To_Go_mm']", "range": [-300, 300]}},
 {
   "joinaggregate": [{
     "op": "count",
     "field": "datum.Align_Distance_To_Go_mm",
     "as": "count_all_Dist"
    }],
    "groupby": ["Current_Platform_Name"]
  },
 
  {
    "joinaggregate": [{
      "op": "count",
      "field": "datum.Align_Distance_To_Go_mm",
      "as": "count_within_Dist"
    }],
    "groupby": ["Current_Platform_Name"]
    },
   {
    "calculate": " (datum.count_within_Dist/datum.count_all_Dist)*100 ",
    "as": "Percent_Compliance"
  }
],

"layer": [
{
"mark": "bar",
"height": {"step": 5},
"encoding": {
  "x": {
    "field": "Percent_Compliance",
    "type": "quantitative",
    "axis": {"title": "Percent_Compliance"}
  },
  "y": {
    "field": "_source.Current_Platform_Name",
    "type": "nominal",
    "axis": {"title": "Platform Name"}
  },
  "color": {
    "field": "_source.Current_Platform_Name",
    "type": "nominal",
    "legend": {"title": "Platform Legend"}
  },
  "shape": {"field": "_source.Current_Platform_Name", "type": "nominal"}
}
}
]

}

here is my code. i dont get why its not working out.

i do prefer using Vega 4.3.0 - so i do seek your advice from the Vega approach instead. Thanks

I think your specs have multiple problems, and I will try to help you find those problems. The main problems are:

  • Transforms that use field do not use datum. You should write _source.Align_Distance_To_Go_mm instead of datum._source.Align_Distance_To_Go_mm in transforms.
  • Transforms are not the same in Vega and Vega-lite- you can't copy and paste from Vega-lite into Vega.

The best way to debug this is to use the browser dev tools and inspect the following two variables:

VEGA_DEBUG.view.data('source_0')
VEGA_DEBUG.view.data('data_0')

In vega-lite, datum refers to data_0, while source_0 is the ES data.

Putting this together, I think this is what you need to write:

"$schema": "https://vega.github.io/schema/vega-lite/v2.json",
...
"transform": [
 {"filter": {"field": "_source.Align_Distance_To_Go_mm", "range": [-300, 300]}},
 {
   "joinaggregate": [{
     "op": "count",
     "field": "_source.Align_Distance_To_Go_mm",
     "as": "count_all_Dist"
    }],
    "groupby": ["_source.Current_Platform_Name"]
  },
  {
    "joinaggregate": [{
      "op": "count",
      "field": "_source.Align_Distance_To_Go_mm",
      "as": "count_within_Dist"
    }],
    "groupby": ["_source.Current_Platform_Name"]
    },
   {
    "calculate": " (datum.count_within_Dist/datum.count_all_Dist)*100 ",
    "as": "Percent_Compliance"
  }
],

hi Wylie, thanks for ur explanation but hold on.

Am i only restricted to using Vega-Lite only for Kibana?

  1. I realised me %timefield% : DateTime was incorrect. The other Kibana instance i borrow the code from happened to have DateTime as a field inside the document. But this one didnt. I've since changed it to Arrival_Time.

  2.     {
         "$schema": "https://vega.github.io/schema/vega-lite/v2.json",
         "mark": "point",
         "data": {
           "url": {
             "%context%": true,
             "%timefield%": "Arrival_Time",
             "index": "nsew*",
             "body": {
               "size": 10000,
               "_source": ["Current_Platform_Name", "Align_Distance_To_Go_mm","Train_Type"]
             }
           },
           "format": {"property": "hits.hits"}
         },
      "transform": [
      {"filter": {"field": "_source.Align_Distance_To_Go_mm", "range": [-300, 300]}},
      {
        "joinaggregate": [{
          "op": "count",
          "field": "_source.Align_Distance_To_Go_mm",
          "as": "count_all_Dist"
         }],
         "groupby": ["_source.Current_Platform_Name"]
       },
       {
         "joinaggregate": [{
           "op": "count",
           "field": "_source.Align_Distance_To_Go_mm",
           "as": "count_within_Dist"
         }],
         "groupby": ["_source.Current_Platform_Name"]
         },
        {
         "calculate": " ( datum.count_within_Dist / datum.count_all_Dist)*100 ",
         "as": "Percent_Compliance"
       }
     ],
         "encoding": {
           "x": {
             "field": "Percent_Compliance",
             "type": "quantitative",
             "axis": {"title": "Percent_Compliance"}
           },
           "y": {
             "field": "_source.Current_Platform_Name",
             "type": "nominal",
             "axis": {"title": "Platform Name"}
           },
           "color": {
             "field": "_source.Current_Platform_Name",
             "type": "nominal",
             "legend": {"title": "Platform Legend"}
           },
           "shape": {"field": "_source.Current_Platform_Name", "type": "nominal"}
         }
       }
    

This my code as of now, taking into acct ur changes. But nothing still comes out.

I entered the debugger, 2 statements, now i can retrieve data frm Elasticsearch since i entered a correct %timefield% but data_0 is still empty.

SUSPICIONS

  1. I know how i want to wrangle my data but based on invalid Transform, am i possibly doing it wrongly?

  2. Would my this use-case be even possible in Vega? I re-iterate it as below:

Find the percentage of the count of values in Align_Distance_To_Go_mm that are within +/-300, grouped by Current Platform Name.

Represent using horizontal bar graph, y-axis Current Platform Name, x-axis Percent Compliance. Graph can be sorted and can show z numbers of either Top % or Bottom %.

I believe your issue is an invalid filter. Try this:

{ "filter": "datum._source.Align_Distance_To_Go_mm >= -300 && datum._source.Align_Distance_To_Go_mm <= 300" },

This visualization should be possible.

nope still didnt work out.

any idea why

Could you post the JSON code and the error messages? Can't read the screenshots

/built_assets/dlls/vendors_0.bundle.dll.js:145 WARN Cannot use a fixed value of "rangeStep" when "autosize" is "fit".
log @ /built_assets/dlls/vendors_0.bundle.dll.js:145
warn @ /built_assets/dlls/vendors_0.bundle.dll.js:145
warn @ /built_assets/dlls/vendors_0.bundle.dll.js:159
(anonymous) @ /built_assets/dlls/vendors_0.bundle.dll.js:159
parseUnitScaleRange @ /built_assets/dlls/vendors_0.bundle.dll.js:159
parseScaleRange @ /built_assets/dlls/vendors_0.bundle.dll.js:159
parseScale @ /built_assets/dlls/vendors_0.bundle.dll.js:159
Model.parseScale @ /built_assets/dlls/vendors_0.bundle.dll.js:159
Model.parse @ /built_assets/dlls/vendors_0.bundle.dll.js:159
compile @ /built_assets/dlls/vendors_0.bundle.dll.js:159
_compileVegaLite @ /bundles/commons.bundle.js:3
_callee2$ @ /bundles/commons.bundle.js:3
l @ /bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:288
(anonymous) @ /bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:288
forEach.e.<computed> @ /bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:288
asyncGeneratorStep @ /bundles/commons.bundle.js:3
_next @ /bundles/commons.bundle.js:3
/built_assets/dlls/vendors_0.bundle.dll.js:145 WARN Ignoring an invalid transform: {"groupby":["_source.Current_Platform_Name"],"joinaggregate":[{"as":"count_all_Dist","field":"_source.Align_Distance_To_Go_mm","op":"count"}]}.
log @ /built_assets/dlls/vendors_0.bundle.dll.js:145
warn @ /built_assets/dlls/vendors_0.bundle.dll.js:145
warn @ /built_assets/dlls/vendors_0.bundle.dll.js:159
(anonymous) @ /built_assets/dlls/vendors_0.bundle.dll.js:159
parseTransformArray @ /built_assets/dlls/vendors_0.bundle.dll.js:159
parseData @ /built_assets/dlls/vendors_0.bundle.dll.js:159
UnitModel.parseData @ /built_assets/dlls/vendors_0.bundle.dll.js:159
Model.parse @ /built_assets/dlls/vendors_0.bundle.dll.js:159
compile @ /built_assets/dlls/vendors_0.bundle.dll.js:159
_compileVegaLite @ /bundles/commons.bundle.js:3
_callee2$ @ /bundles/commons.bundle.js:3
l @ /bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:288
(anonymous) @ /bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:288
forEach.e.<computed> @ /bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:288
asyncGeneratorStep @ /bundles/commons.bundle.js:3
_next @ /bundles/commons.bundle.js:3
/built_assets/dlls/vendors_0.bundle.dll.js:145 WARN Ignoring an invalid transform: {"groupby":["_source.Current_Platform_Name"],"joinaggregate":[{"as":"count_within_Dist","field":"_source.Align_Distance_To_Go_mm","op":"count"}]}.
log @ /built_assets/dlls/vendors_0.bundle.dll.js:145
warn @ /built_assets/dlls/vendors_0.bundle.dll.js:145
warn @ /built_assets/dlls/vendors_0.bundle.dll.js:159
(anonymous) @ /built_assets/dlls/vendors_0.bundle.dll.js:159
parseTransformArray @ /built_assets/dlls/vendors_0.bundle.dll.js:159
parseData @ /built_assets/dlls/vendors_0.bundle.dll.js:159
UnitModel.parseData @ /built_assets/dlls/vendors_0.bundle.dll.js:159
Model.parse @ /built_assets/dlls/vendors_0.bundle.dll.js:159
compile @ /built_assets/dlls/vendors_0.bundle.dll.js:159
_compileVegaLite @ /bundles/commons.bundle.js:3
_callee2$ @ /bundles/commons.bundle.js:3
l @ /bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:288
(anonymous) @ /bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:288
forEach.e.<computed> @ /bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:288
asyncGeneratorStep @ /bundles/commons.bundle.js:3
_next @ /bundles/commons.bundle.js:3
/built_assets/dlls/vendors_0.bundle.dll.js:145 WARN Cannot use a fixed value of "rangeStep" when "autosize" is "fit".
log @ /built_assets/dlls/vendors_0.bundle.dll.js:145
/built_assets/dlls/vendors_0.bundle.dll.js:145 WARN Ignoring an invalid transform: {"groupby":["_source.Current_Platform_Name"],"joinaggregate":[{"as":"count_all_Dist","field":"_source.Align_Distance_To_Go_mm","op":"count"}]}.
log @ /built_assets/dlls/vendors_0.bundle.dll.js:145
/built_assets/dlls/vendors_0.bundle.dll.js:145 WARN Ignoring an invalid transform: {"groupby":["_source.Current_Platform_Name"],"joinaggregate":[{"as":"count_within_Dist","field":"_source.Align_Distance_To_Go_mm","op":"count"}]}.
log @ /built_assets/dlls/vendors_0.bundle.dll.js:145
/built_assets/dlls/vendors_0.bundle.dll.js:145 WARN Cannot use a fixed value of "rangeStep" when "autosize" is "fit".
log @ /built_assets/dlls/vendors_0.bundle.dll.js:145
/built_assets/dlls/vendors_0.bundle.dll.js:145 WARN Ignoring an invalid transform: {"groupby":["_source.Current_Platform_Name"],"joinaggregate":[{"as":"count_all_Dist","field":"_source.Align_Distance_To_Go_mm","op":"count"}]}.
log @ /built_assets/dlls/vendors_0.bundle.dll.js:145
/built_assets/dlls/vendors_0.bundle.dll.js:145 WARN Ignoring an invalid transform: {"groupby":["_source.Current_Platform_Name"],"joinaggregate":[{"as":"count_within_Dist","field":"_source.Align_Distance_To_Go_mm","op":"count"}]}.
log @ /built_assets/dlls/vendors_0.bundle.dll.js:145
// Transforms that use field do not use datum (ie joinaggregate)
// But Transforms like calculate do

{
    "$schema": "https://vega.github.io/schema/vega-lite/v2.json",
    "mark": "point",
    "data": {
      "url": {
        "%context%": true,
        "%timefield%": "Arrival_Time",
        "index": "nsew*",
        "body": {
          "size": 10000,
          "_source": ["Current_Platform_Name", "Align_Distance_To_Go_mm","Train_Type"]
        }
      },
      "format": {"property": "hits.hits"}
    },
 "transform": [
 { "filter": "datum._source.Align_Distance_To_Go_mm >= -300 && datum._source.Align_Distance_To_Go_mm <= 300" },
 {
   "joinaggregate": [{
     "op": "count",
     "field": "_source.Align_Distance_To_Go_mm",
     "as": "count_all_Dist"
    }],
    "groupby": ["_source.Current_Platform_Name"]
  },
  {
    "joinaggregate": [{
      "op": "count",
      "field": "_source.Align_Distance_To_Go_mm",
      "as": "count_within_Dist"
    }],
    "groupby": ["_source.Current_Platform_Name"]
    },
   {
    "calculate": " ( datum.count_within_Dist / datum.count_all_Dist)*100 ",
    "as": "Percent_Compliance"
  }
],
    "encoding": {
      "x": {
        "field": "Percent_Compliance",
        "type": "quantitative",
        "axis": {"title": "Percent_Compliance"}
      },
      "y": {
        "field": "_source.Current_Platform_Name",
        "type": "nominal",
        "axis": {"title": "Platform Name"}
      },
      "color": {
        "field": "_source.Current_Platform_Name",
        "type": "nominal",
        "legend": {"title": "Platform Legend"}
      },
      "shape": {"field": "_source.Current_Platform_Name", "type": "nominal"}
    }
  }

Okay, here is my recommendation for debugging these transformations- I think they are the cause of the issue, and you should start by removing all transformations and adding them one by one. After each transformation, inspect the VEGA_DEBUG.view.data('data_0') to see what it looks like, then iterate. Also, VEGA_DEBUG.view._runtime.data contains all the data that is generated internally, but it's a little less useful.

If you are trying to do something like "Count all the documents with this particular condition" then I actually think you should be using Elasticsearch filters instead of Vega filters.

If you are trying to do something like "Count all the documents with this particular condition" then I actually think you should be using Elasticsearch filters instead of Vega filters.

yes i actually intended this. But Vega itself has a count operator - so why isit not able perform my intent?

btw What is meant by Elasticsearch filters? or do u mean Kibana filters?

Do i need to do some ES queries? Coz as of now, i create new Vega visualisation and then enter the json from there.

im not sure how to manually go in and implement my own ES query whilst preparing Kibana vis. Im still pretty frontend guy. :frowning:

Vega does have a count operator, but your visualization will be much faster if you use the Elasticsearch filters aggregation instead of looking at only 10k docs. Here is how to create a filters aggregation: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-filters-aggregation.html

Here is an example that is similar to what you're doing:

{
  $schema: https://vega.github.io/schema/vega-lite/v2.json
  data: {
    url: {
      %context%: true
      %timefield%: @timestamp
     index: metricbeat-*
     body: {
        aggs: {
          categories: {
            terms: { field: "some_category_field", size: 20 }
            aggs: {
              in_or_out: {
                filters: {
                  filters: {
                    in: {
                      range: {
                        "some_number_field": {
                          "gte": -300,
                          "lte": 300
                        }
                      }
                    },
                    out: {
                      bool: {
                        must_not: {
                          range: {
                            "some_number_field": {
                              "gte": -300,
                              "lte": 300
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
        size: 0
      }
    }

    format: {property: "aggregations.categories.buckets"}
  }

  transform: [
    {
      calculate: "datum.in_or_out.buckets.in.doc_count / datum.in_or_out.buckets.out.doc_count",
      as: "percent"
    }
  ]

  mark: bar

  encoding: {
    y: {
      field: "key"
      type: "ordinal"
      sort: {
        field: "percent",
        order: "descending"
        op: "max"
      }
    }
    x: {
      field: "percent"
      type: quantitative
      format: ".2%"
    }
  }
}

i managed to substitute the values in as accordingly.

{
  $schema: https://vega.github.io/schema/vega-lite/v2.json
  data: {
    url: {
      %context%: true
      %timefield%: "Arrival_Time"
     index: nsew*
     body: {
        aggs: {
          categories: {
            terms: { field: "Align_Distance_To_Go_mm", size: 10000 }
            aggs: {
              in_or_out: {
                filters: {
                  filters: {
                    in: {
                      range: {
                        "Align_Distance_To_Go_mm": {
                          "gte": -300,
                          "lte": 300
                        }
                      }
                    },
                    out: {
                      bool: {
                        must_not: {
                          range: {
                            "Align_Distance_To_Go_mm": {
                              "lte": -300,
                              "gte": 300
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
        size: 0
      }
    }

    format: {property: "aggregations.categories.buckets"}
  }

  transform: [
    {
      calculate: "datum.in_or_out.buckets.in.doc_count / datum.in_or_out.buckets.out.doc_count",
      as: "Percent Compliance"
    }
  ]

  mark: bar

  encoding: {
    y: {
      field: "Current_Platform_Name"
      type: "ordinal"
      sort: {
        field: "percent",
        order: "descending"
        op: "max"
      }
      "axis": {"title": "Current Platform Name"}
    }
    x: {
      field: "Percent Compliance"
      type: quantitative
      format: ".2%"
    }
  }
}

The output is in direction but still not quite there.

I eventually desire this:

body: {
        aggs: {
          categories: {
            terms: { field: "Align_Distance_To_Go_mm", size: 10000 }
            aggs: {
              in_or_out: {
                filters: {
                  filters: {
                    in: {
                      range: {
                        "Align_Distance_To_Go_mm": {
                          "gte": -300,
                          "lte": 300
                        }
                      }
                    },
                    out: {
                      bool: {
                        must_not: {
                          range: {
                            "Align_Distance_To_Go_mm": {
                              "lte": -300,
                              "gte": 300
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
        size: 0
      }
    }

    format: {property: "aggregations.categories.buckets"}
  }

so this block of code isit an ES query integrated into the Vega ?

Yes, it's an ES search query.