Vega transform-stack not calculating correctly


(Mathew Thekkekara) #1

Hi There, I got a problem,

I am trying to create a chart with VEGA. I want in the x axis the time and in the y axis the number of doc_count. And ofcourse for coolnes i wanted the data in y-axis stacked by the continent_code.
The stacking creates a y0 and y1 for me, this works fine. But this y0 should start for a new point on the x from 0, but it is having an offset from the last point on x.
Let me show this in image1:


On the 6 elm in the table array there is a new date so the y0 should be 0 but it is taking the value from table[5].y1
And this follows that my chart looks a bit stupid. Let me show you that in image 2:

And this is my code :

{
  "$schema": "https://vega.github.io/schema/vega/v3.json",
  "padding": 5,
  "data": [
    {
      "name": "table",
      "url": {
        "%context%": true,
        "%timefield%": "@timestamp",
        "index": "demo_logs_server*",
        "body": {
          "size": 0,
          "aggs": {
            "logs_time_country": {
              "composite": {
                "size": 50,
                "sources": [
                  {
                    "time": {
                      "date_histogram": {
                        "field": "@timestamp",
                        "interval": {"%autointerval%":200},
                        "format": "dd-MM"
                      }
                    }
                  },
                  {
                    "country": {
                      "terms": {
                        "field": "geoip.continent_code.keyword"
                      }
                    }
                  }
                ]
              }
            }
          }
        }
      },
      "format": {
        "property": "aggregations.logs_time_country.buckets"
      },
      "transform": [
        {
          "type": "stack",
          "groupby": ["time"],
          "sort": {"field":"country"},
          "field": "doc_count"
        }
      ]
    }
  ],
  "scales": [
    {
      "name": "x",
      "type": "point",
      "range": "width",
      "domain": {"data": "table", "field": "key.time"}
    },
    {
      "name": "y",
      "type": "linear",
      "range": "height",
      "nice": true, "zero": true,
      "domain": {"data": "table", "field": "y1"}
    },
    {
      "name": "color",
      "type": "ordinal",
      "range": "category",
      "domain": {"data": "table", "field": "key.country"}
    }
  ],
  "axes": [
    {"orient": "bottom", "scale": "x", "zindex": 1},
    {"orient": "left", "scale": "y", "zindex": 1}
  ],
  "marks": [
    {
      "type": "group",
      "from": {
        "facet": {
          "name": "series",
          "data": "table",
          "groupby": "key.country"
        }
      },
      "marks": [
        {
          "type": "area",
          "from": {"data": "series"},
          "encode": {
            "enter": {
              "interpolate": {"value": "linear"},
              "x": {"scale": "x", "field": "key.time"},
              "y": {"scale": "y", "field": "y0"},
              "y2": {"scale": "y", "field": "y1"},
              "fill": {"scale": "color", "field": "key.country"}
            },
            "update": {
              "fillOpacity": {"value": 1}
            },
            "hover": {
              "fillOpacity": {"value": 0.5}
            }
          }
        }
      ]
    }
  ]
}

I think the mistake is somewhere in the transform-stack cause this is the part which is calculating the y0 and y1.

Will appriciate any kind of help
Thanks folks


(Brandon Kobel) #2

Hey @thekm1, can you share the data returned from Elasticsearch which the vega visualization uses in a github gist?


(Mathew Thekkekara) #3

Hi Brandon, Thanks for answering.
I just took away the sort attribute from transform-stack
now its like

  "transform": [
    {
      "type": "stack",
      "groupby": ["time"],
      "field": "doc_count"
    }
  ]

This made a better graph.. got till the stacking problem, but don't have have the blue areas that's crossing the graph as they like...
see image :


And i reduced the size of the composite aggr from 50 to 33. this gives me a shorter response that i can publish in gist.
my new aggr:

"body": {
  "size": 0,
  "aggs": {
    "logs_time_country": {
      "composite": {
        "size": 33,
        "sources": [
          {
            "time": {
              "date_histogram": {
                "field": "@timestamp",
                "interval": {"%autointerval%":200},
                "format": "dd-MM"
              }
            }
          },
          {
            "country": {
              "terms": {
                "field": "geoip.continent_code.keyword"
              }
            }
          }
        ]
      }
    }
  }
}

And here is the response from elastic search that you asked for :slight_smile:

"[
  {
    "key": {
      "time": "31-03",
      "country": "AF"
    },
    "doc_count": 1,
    "y0": 0,
    "y1": 1
  },
  {
    "key": {
      "time": "31-03",
      "country": "AS"
    },
    "doc_count": 18,
    "y0": 1,
    "y1": 19
  },
  {
    "key": {
      "time": "31-03",
      "country": "EU"
    },
    "doc_count": 40,
    "y0": 19,
    "y1": 59
  },
  {
    "key": {
      "time": "31-03",
      "country": "NA"
    },
    "doc_count": 183,
    "y0": 59,
    "y1": 242
  },
  {
    "key": {
      "time": "31-03",
      "country": "OC"
    },
    "doc_count": 4,
    "y0": 242,
    "y1": 246
  },
  {
    "key": {
      "time": "31-03",
      "country": "SA"
    },
    "doc_count": 9,
    "y0": 246,
    "y1": 255
  },
  {
    "key": {
      "time": "01-04",
      "country": "AF"
    },
    "doc_count": 14,
    "y0": 255,
    "y1": 269
  },
  {
    "key": {
      "time": "01-04",
      "country": "AS"
    },
    "doc_count": 993,
    "y0": 269,
    "y1": 1262
  },
  {
    "key": {
      "time": "01-04",
      "country": "EU"
    },
    "doc_count": 679,
    "y0": 1262,
    "y1": 1941
  },
  {
    "key": {
      "time": "01-04",
      "country": "NA"
    },
    "doc_count": 1329,
    "y0": 1941,
    "y1": 3270
  },
  {
    "key": {
      "time": "01-04",
      "country": "OC"
    },
    "doc_count": 59,
    "y0": 3270,
    "y1": 3329
  },
  {
    "key": {
      "time": "01-04",
      "country": "SA"
    },
    "doc_count": 26,
    "y0": 3329,
    "y1": 3355
  },
  {
    "key": {
      "time": "01-04",
      "country": "AF"
    },
    "doc_count": 23,
    "y0": 3355,
    "y1": 3378
  },
  {
    "key": {
      "time": "01-04",
      "country": "AS"
    },
    "doc_count": 566,
    "y0": 3378,
    "y1": 3944
  },
  {
    "key": {
      "time": "01-04",
      "country": "EU"
    },
    "doc_count": 1099,
    "y0": 3944,
    "y1": 5043
  },
  {
    "key": {
      "time": "01-04",
      "country": "NA"
    },
    "doc_count": 2248,
    "y0": 5043,
    "y1": 7291
  },
  {
    "key": {
      "time": "01-04",
      "country": "OC"
    },
    "doc_count": 24,
    "y0": 7291,
    "y1": 7315
  },
  {
    "key": {
      "time": "01-04",
      "country": "SA"
    },
    "doc_count": 59,
    "y0": 7315,
    "y1": 7374
  },
  {
    "key": {
      "time": "02-04",
      "country": "AF"
    },
    "doc_count": 15,
    "y0": 7374,
    "y1": 7389
  },
  {
    "key": {
      "time": "02-04",
      "country": "AS"
    },
    "doc_count": 638,
    "y0": 7389,
    "y1": 8027
  },
  {
    "key": {
      "time": "02-04",
      "country": "EU"
    },
    "doc_count": 565,
    "y0": 8027,
    "y1": 8592
  },
  {
    "key": {
      "time": "02-04",
      "country": "NA"
    },
    "doc_count": 1151,
    "y0": 8592,
    "y1": 9743
  },
  {
    "key": {
      "time": "02-04",
      "country": "OC"
    },
    "doc_count": 57,
    "y0": 9743,
    "y1": 9800
  },
  {
    "key": {
      "time": "02-04",
      "country": "SA"
    },
    "doc_count": 21,
    "y0": 9800,
    "y1": 9821
  },
  {
    "key": {
      "time": "02-04",
      "country": "AF"
    },
    "doc_count": 20,
    "y0": 9821,
    "y1": 9841
  },
  {
    "key": {
      "time": "02-04",
      "country": "AS"
    },
    "doc_count": 1517,
    "y0": 9841,
    "y1": 11358
  },
  {
    "key": {
      "time": "02-04",
      "country": "EU"
    },
    "doc_count": 1925,
    "y0": 11358,
    "y1": 13283
  },
  {
    "key": {
      "time": "02-04",
      "country": "NA"
    },
    "doc_count": 1482,
    "y0": 13283,
    "y1": 14765
  },
  {
    "key": {
      "time": "02-04",
      "country": "OC"
    },
    "doc_count": 56,
    "y0": 14765,
    "y1": 14821
  },
  {
    "key": {
      "time": "02-04",
      "country": "SA"
    },
    "doc_count": 39,
    "y0": 14821,
    "y1": 14860
  },
  {
    "key": {
      "time": "03-04",
      "country": "AF"
    },
    "doc_count": 61,
    "y0": 14860,
    "y1": 14921
  },
  {
    "key": {
      "time": "03-04",
      "country": "AS"
    },
    "doc_count": 1814,
    "y0": 14921,
    "y1": 16735
  },
  {
    "key": {
      "time": "03-04",
      "country": "EU"
    },
    "doc_count": 2490,
    "y0": 16735,
    "y1": 19225
  }
]"

Thanks a lot for helping
regards thekm1


(kulkarni) #4

@nyuriks - can u plz help ?

Cheers
Rashmi


(Mathew Thekkekara) #5

Thankyou rashmi

I got it. It was a mistake that i made in the transform options. As i declared the field groupby and sort i hadn't give the accessing of the array. cause the time and country that i given before is in the key array. so need to access the array.
So i think my problem is solved.

Thankyou to everybody for the support. =)
The result :


(kulkarni) #6

Great . Thanks for posting the solution you arrived at.

Cheers
Rashmi


(system) #7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.