Vega Pie Chart on dashboard

theflyingcarpet · January 26, 2019, 9:57am

I'm using vega to draw a pie chart in a dashboard to get around the issues with visualisations from different indices in a dashboard (i.e. select anything in one index's viz and it filters out everything in the other index's viz). Note this is because ignoreFilterIfFieldNotInIndex does not work for me in 6.5.1 (see https://github.com/elastic/kibana/issues/28784). I'm trying to use the code from this post https://groups.google.com/forum/#!topic/vega-js/bkwVdJyWrDI as it has several nice features.
I have a working query which returns the raw aggregation data I need to display (and fires fine in the context of the vega code) but I'm really struggling to get the data out of the response and into the viz. I also can't seem to get the vega debug to view the raw data response either. Any help massively appreciated!

Here's the working aggregated response I'm getting

{"took":0,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":6040402,"max_score":0.0,"hits":},"aggregations":{"featurelist":{"doc_count_error_upper_bound":0,"sum_other_doc_count":5593771,"buckets":[{"key":"CELA1","doc_count":130520},{"key":"ZNF717","doc_count":86363},{"key":"HLA-DRB5","doc_count":82724},{"key":"MUC3A","doc_count":77286},{"key":"SAMD1","doc_count":66276}]}}}`

And here is the vega code that I've simplified it to a single pie chart and just trying to display the returned keys by the count in the chart. I don't understand what transforms/projection I need to get it joined up.

{
  "$schema": "https://vega.github.io/schema/vega/v3.0.json",
  "autosize": "fit",
  "data": [
    {
      "name": "data_table",
      "url": {
        "index": "tindx",
        "body": {
          "size": 0,
          "query": {"bool": {"filter": [{"match_all": {}}]}},
          "aggs": {
            "featurelist": {
              "terms": {
                "field": "feature.keyword",
                "size": 5,
                "order": {"_count": "desc"}
              }
            }
          }
        }
      },
      "format": {"property": "aggregations.featurelist.buckets"},
      "transform": [
        {	"type": "collect",
          "sort": {
            "field": ["doc_count"],
            "order": ["descending"]
          }
        }
      ]
    },
    {
      "name": "data_table_pie_inner",
      "source": "data_table",
      "transform": [
        {
          "type": "aggregate",
          "groupby": ["key"],
          "fields": ["doc_count"],
          "ops": ["sum"],
          "as": ["ff_sum_percent"]
        },
        {
          "type": "pie",
          "field": "ff_sum_percent",
          "as": ["ff_inner_startAngle", "ff_inner_endAngle"]
        }
      ]
    }
  ],
  "scales": [
    {
      "name": "scale_color",
      "type": "ordinal",
      "range": {"scheme": "category10"},
      "domain": {"data": "data_table", "field": "key"}
    }
  ],
  "marks": [
    {
      "name": "mark_inner_ring",
      "type": "arc",
      "from": {"data": "data_table_pie_inner"},
      "encode": {
        "enter": {
          "x": {"signal": "width / 2"},
          "y": {"signal": "height / 2"},
          "fill": {"scale": "scale_color", "field": "key"},
          "fillOpacity": {"value": 0.8},
          "stroke": {"value": "white"},
          "startAngle": {"field": "ff_inner_startAngle"},
          "endAngle": {"field": "ff_inner_endAngle"},
          "innerRadius": {"value": 0},
          "outerRadius": {"value": 100},
          "tooltip": {
            "signal": "datum['key'] + ': ' + datum['doc_count'] + ' (' + datum['ff_sum_percent'] + '%)'"
          }
        }
      }
    },
    {
      "name": "mark_text_os_version",
      "type": "text",
      "from": {"data": "data_table_pie_inner"},
      "encode": {
        "enter": {
          "text": {"signal": "datum['key'] == '' ? '' : datum['key']"},
          "x": {"signal": "width / 2"},
          "y": {"signal": "height / 2"},
          "radius": {"value": 115},
          "theta": {
            "signal": "(datum['ff_inner_startAngle'] + datum['ff_inner_endAngle'])/2"
          },
          "fill": {"value": "black"},
          "font": {"value": "Helvetica"},
          "fontSize": {"value": 12},
          "align": {"value": "center"},
          "baseline": {"value": "middle"}
        }
      }
    }
  ],
  "legends": [
    {
      "fill": "scale_color",
      "title": "Feature",
      "orient": "right",
      "encode": {
        "symbols": {"enter": {"fillOpacity": {"value": 0.5}}},
        "labels": {"update": {"text": {"field": "value"}}}
      }
    }
  ]
}

Update: I've got a bit further now and the pie is being drawn (amended code above). Now the reported numbers on tooltip are wrong where I'd like to display a % calculation. I need the total doc count for that which is the "hits.total" and is outside the aggregation result. How can I access that and use it in the percentage calculation? And still not getting the vega debug to work. But getting closer...

theflyingcarpet · January 28, 2019, 12:52pm

There must be some simple way of getting the hits.total value into the tooltip percentage calculation but I've been at it for hours and can't see it. This latest version issues an additional query just for the hits total and tries to merge the two data sources. The problem here is the second data source is appended to the first and I still can't get at the total (which has been added as the last array element)

{
  "$schema": "https://vega.github.io/schema/vega/v3.0.json",
  "autosize": "fit",
  "data": [
    {
      "name": "data_table",
      "url": {
        "index": "tindx",
        "body": {
          "size": 0,
          "query": {"bool": {"filter": [{"match_all": {}}]}},
          "aggs": {
            "featurelist": {
              "terms": {
                "field": "feature.keyword",
                "size": 10,
                "order": {"_count": "desc"}
              }
            }
          }
        }
      },
      "format": {"property": "aggregations.featurelist.buckets"},
      "transform": [
        {	"type": "collect",
          "sort": {
            "field": ["doc_count"],
            "order": ["descending"]
          }
        }
      ]
    },
    {
      "name": "data_table_pie_inner",
      "source": "data_table",
      "transform": [
        {
          "type": "aggregate",
          "groupby": ["key"],
          "fields": ["doc_count"],
          "ops": ["sum"],
          "as": ["ff_sum_count"]
        },
        {
          "type": "pie",
          "field": "ff_sum_count",
          "as": ["ff_inner_startAngle", "ff_inner_endAngle"]
        }
      ]
    },
    {
      "name": "hits_table",
      "url": {
        "index": "tidx",
        "body": {
          "size": 0,
          "query": {"bool": {"filter": [{"match_all": {}}]}}
        }
      },
      "format": {"property": "hits"},
      "transform": [
        {
          "type": "collect",
          "sort": {"field": "total", "order": ["descending"]}
        }
      ]
    },
    {
      "name": "hits_table_total",
      "source": "hits_table",
      "transform": [
        {
          "type": "aggregate",
          "fields": ["total"],
          "ops": ["sum"],
          "as": ["ff_total_count"]
        }
      ]
    },
    {
      "name": "data_table_combo",
      "source": ["data_table_pie_inner","hits_table_total"],
      "transform": [
        {"type": "formula", "as": "overall_total", "expr": "'parent.ff_total_count'"}
      ]
    }
  ],
  "scales": [
    {
      "name": "scale_color",
      "type": "ordinal",
      "range": {"scheme": "category10"},
      "domain": {"data": "data_table", "field": "key"}
    }
  ],
  "marks": [
    {
      "name": "mark_inner_ring",
      "type": "arc",
      "from": {"data": "data_table_combo"},
      "encode": {
        "enter": {
          "x": {"signal": "width / 2"},
          "y": {"signal": "height / 2"},
          "fill": {"scale": "scale_color", "field": "key"},
          "fillOpacity": {"value": 0.8},
          "stroke": {"value": "white"},
          "startAngle": {"field": "ff_inner_startAngle"},
          "endAngle": {"field": "ff_inner_endAngle"},
          "innerRadius": {"value": 0},
          "outerRadius": {"value": 100},
          "tooltip": {
            "signal": "datum['key'] + ': count ' + datum['ff_sum_count'] + ' (' + datum['overall_total'] + '%)'"
        }
      }
      }
    }
  ],
  "legends": [
    {
      "fill": "scale_color",
      "title": "Top 10 Features",
      "orient": "right",
      "encode": {
        "symbols": {"enter": {"fillOpacity": {"value": 0.5}}},
        "labels": {"update": {"text": {"field": "value"}}}
      }
    }
  ]
}

Larry_Gregory · January 28, 2019, 3:26pm

@nyuriks

I can get @theflyingcarpet's pie chart to render, but I'm not able to make any progress on getting the tooltip to render what they're looking for. Do you have any suggestions?

Larry_Gregory · January 28, 2019, 4:45pm

@theflyingcarpet,

I managed to get the total hits down into the data set by modifying your second example.

I added a key to your hits_table_total datasource transform, so that I could perform a lookup on it later.
I changed the transform in data_table_combo to be a lookup, and I used that to grab the total from step 1 above:

{
      name: hits_table_total
      source: hits_table
      transform: [
        {
          type: aggregate
          fields: ["total"]
          ops: ["sum"]
          as: ["ff_total_count"]
          key: ff_total_count_key
        }
      ]
    }
    {
      name: data_table_combo
      source: ["data_table_pie_inner"]
      transform: [
        {
          type: lookup
          from: hits_table_total
          key: ff_total_count_key
          fields: ["ff_total_count"]
          as: ["overall_total"]
        }
      ]
    }

Then, you can get access to the value by using datum.overall_total.ff_total_count:

{
      name: mark_inner_ring
      type: arc
      from: {data: "data_table_combo"}
      encode: {
        enter: {
          x: {signal: "width / 2"}
          y: {signal: "height / 2"}
          fill: {scale: "scale_color", field: "key"}
          fillOpacity: {value: 0.8}
          stroke: {value: "white"}
          startAngle: {field: "ff_inner_startAngle"}
          endAngle: {field: "ff_inner_endAngle"}
          innerRadius: {value: 0}
          outerRadius: {value: 100}
          tooltip: {signal: "datum['key'] + ': count ' + datum['ff_sum_count'] + ' (' + datum.overall_total.ff_total_count + '%)'"}
        }
      }
    }

So the full example looks something like this:

{
  $schema: https://vega.github.io/schema/vega/v3.0.json
  autosize: fit
  data: [
    {
      name: data_table
      url: {
        index: tindx
        body: {
          size: 0
          query: {
            bool: {
              filter: [
                {
                  match_all: {}
                }
              ]
            }
          }
          aggs: {
            featurelist: {
              terms: {
                field: entity_id
                size: 10
                order: {_count: "desc"}
              }
            }
          }
        }
      }
      format: {property: "aggregations.featurelist.buckets"}
      transform: [
        {
          type: collect
          sort: {
            field: ["doc_count"]
            order: ["descending"]
          }
        }
      ]
    }
    {
      name: data_table_pie_inner
      source: data_table
      transform: [
        {
          type: aggregate
          groupby: ["key"]
          fields: ["doc_count"]
          ops: ["sum"]
          as: ["ff_sum_count"]
        }
        {
          type: pie
          field: ff_sum_count
          as: ["ff_inner_startAngle", "ff_inner_endAngle"]
        }
      ]
    }
    {
      name: hits_table
      url: {
        index: tindx
        body: {
          size: 0
          query: {
            bool: {
              filter: [
                {
                  match_all: {}
                }
              ]
            }
          }
        }
      }
      format: {property: "hits"}
      transform: [
        {
          type: collect
          sort: {
            field: total
            order: ["descending"]
          }
        }
      ]
    }
    {
      name: hits_table_total
      source: hits_table
      transform: [
        {
          type: aggregate
          fields: ["total"]
          ops: ["sum"]
          as: ["ff_total_count"]
          key: ff_total_count_key
        }
      ]
    }
    {
      name: data_table_combo
      source: ["data_table_pie_inner"]
      transform: [
        {
          type: lookup
          from: hits_table_total
          key: ff_total_count_key
          fields: ["ff_total_count"]
          as: ["overall_total"]
        }
      ]
    }
  ]
  scales: [
    {
      name: scale_color
      type: ordinal
      range: {scheme: "category10"}
      domain: {data: "data_table", field: "key"}
    }
  ]
  marks: [
    {
      name: mark_inner_ring
      type: arc
      from: {data: "data_table_combo"}
      encode: {
        enter: {
          x: {signal: "width / 2"}
          y: {signal: "height / 2"}
          fill: {scale: "scale_color", field: "key"}
          fillOpacity: {value: 0.8}
          stroke: {value: "white"}
          startAngle: {field: "ff_inner_startAngle"}
          endAngle: {field: "ff_inner_endAngle"}
          innerRadius: {value: 0}
          outerRadius: {value: 100}
          tooltip: {signal: "datum['key'] + ': count ' + datum['ff_sum_count'] + ' (' + datum.overall_total.ff_total_count + '%)'"}
        }
      }
    }
  ]
  legends: [
    {
      fill: scale_color
      title: Top 10 Features
      orient: right
      encode: {
        symbols: {
          enter: {
            fillOpacity: {value: 0.5}
          }
        }
        labels: {
          update: {
            text: {field: "value"}
          }
        }
      }
    }
  ]
}

theflyingcarpet · January 29, 2019, 10:15am

Hey thanks @Larry_Gregory! That lookup was the missing piece. And now I just needed to actually do the % calculation instead of just pretending. here's the signal line if anyone else is interested.

"signal": "datum['key'] + ': count ' + datum['ff_sum_count'] + ' (' + format((datum.ff_sum_count/datum.overall_total.ff_total_count) * 100, '.2f') + '%)'"

system · February 26, 2019, 10:23am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Put kibanaFilter while redirecting to other dashboard using Vega Kibana vega	4	302	January 25, 2024
Vega Pie Chart not Rendering Kibana vega	5	1121	March 19, 2019
Vega visualizations not getting updated Kibana	2	314	October 14, 2020
Frozen Indices on Vega Kibana	3	455	September 17, 2019
Vega filter with controls in dashboard Kibana	2	933	June 1, 2020

Vega Pie Chart on dashboard

Related topics