Issue Vega to plot timeseries data in longer time range

[f2020 (felicia halim)]

Hi,

Currently we are facing an issue when plot the Vega chart in longer time range (1 year) , the recent data not shown. For example: if the time range set to 1 year such as from 9/18/2019 to 9/18/2020. The chart only shows 9/7/2020 (although there is data until 9/18/2020).

Below is the chart configuration, index A has 80,000 data and it has this issue.
But when I change to index B that only has 13,000 data the charts seems can show the data points properly.
So I assumed this issue due to number of data points. Is there any maximum number of points for Vega scatter plot?
Is there any workaround regarding this?

Thank you.

> {
>   "$schema": "https://vega.github.io/schema/vega-lite/v2.json",
>   "mark": "point",
>   "data": {
>     "url": {
>       "%context%": true,
>       "%timefield%": "datetime",
>       "index": "A",
>       "body": {},
>       "size": 10000,
>       "scroll": "3s"
>     },
>     "format": {"property": "hits.hits"}
>   },
>   "transform": [
>     {
>       "calculate": "datum._source['datetime']",
>       "as": "time"
>     },
>     {
>       "calculate": "datum._source['valueField']",
>       "as": "valueField"
>     },
>     {"calculate": "0", "as": "lower_limit"},
>     {"calculate": "26", "as": "upper_limit"},
>     {
>       "calculate": "datum._source",
>       "as": "data_source"
>     }
>   ],
>   "layer": [
>     {
>       "mark": "rule",
>       "encoding": {
>         "y": {"field": "lower_limit", "type": "quantitative"},
>         "color": {"value": "#e45755"}
>       }
>     },
>     {
>       "mark": "rule",
>       "encoding": {
>         "y": {"field": "upper_limit", "type": "quantitative"},
>         "color": {"value": "#e45755"}
>       }
>     },
>   {
>       "mark": "point",
>       "encoding": {
>         "x": {"field": "time", "type": "temporal", "timeUnit": "yearmonthdatehoursminutes", "axis": {"title": false}},
>         "y": {
>           "field": "valueField",
>           "type": "quantitative",
>           "axis": {"title": "valueField"},
>           "scale": {"zero": false}
>         },
>         "color": {"field": "data_source[categoryField]"},
>         tooltip: [
>           {field: "time", "type": "temporal", title: "time","timeUnit": "yearmonthdatehoursminutes"},
>           {field: "valueField", title: "value"}
>         ]
>       }
>     }
>     
>   ]
> }

[wylie (Wylie Conlon)]

If you want to fetch this much data, you need to configure Elasticsearch above the default limit of 10k results using this setting. It's not really recommended, but depending on how powerful your servers are, it can be done.

I strongly recommend that you avoid returning the entire _source field from the query. It looks like you're returning indexed fields, so I recommend running this query instead:

url: {
  %context%: true
  %timefield%: datetime
  index: A
  body: {
    _source: false
    docvalue_fields: ["datetime", "valueField", "categoryField"]
  }
  size: 10000 // or whatever you set it to
}

The scroll parameter you're sending on the query is not doing anything. You can read the docs, but this can't be done in Kibana Vega. https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#scroll-search-results