Kibana Vega-lite data unlimited size

[AditiKhalatkar (Aditi)]

Hi,

I am using an index to fetch data into vega-lite visualization object. My query is how can I fetch the data when it crosses 10000 docs. Is there a way to do that? May be use Scroll API. Not so sure upon that. Kindly guide.

"data": {
 "url": {
  %context%: true,
  "size":10000,
  "index":index_name
  },
   "format": {
  "property": "hits.hits"
}
  },

If I give size more than 10000, then I get this error,

My aim is to bring all the results from the dataset.

Thanks!

[Brandon_Kobel]

@AditiKhalatkar as far as I'm aware, there isn't a way to use the scroll API to return more results. This likely isn't something you should be doing anyway, and I'd suggest using an aggregation to summarize the data which you'll be plotting, as otherwise you're going to take a pretty nasty performance hit.

[AditiKhalatkar (Aditi)]

Hi Brandon,

Thanks for the advice. I’ll try to narrow down the data through aggregation. I think I tried doing similar on console. And couldn’t figure out a way to do nested query. Anyways, may be I can post my query and get some help upon that.

Alternate approach #1

Do you think creating an alias for this particular aggregation and using that alias as index will be helpful?

Alternate approach #2

Trying for a nested aggregation in Vega-lite?

Kindly guide.

[Brandon_Kobel]

@AditiKhalatkar

Alternate approach #1

Do you think creating an alias for this particular aggregation and using that alias as index will be helpful?

I'm not aware of any facility in Elasticsearch which would make this possible. Would you mind elaborating upon this proposed solution?

Alternate approach #2
Trying for a nested aggregation in Vega-lite?

Vega should support the full Elasticsearch search DSL, so any query you can create using Console you can use within Vega.

[AditiKhalatkar (Aditi)]

I'm not aware of any facility in Elasticsearch which would make this possible. Would you mind elaborating upon this proposed solution?

If I can create an aggregation in an alias may be and use that index.

Aliases API | Elasticsearch Guide [8.11] | Elastic

For instance, suppose there are 10*10000 records in my index.Instead of using this index, I create an alias with aggregation to fetch only 1000 records. Use this alias index in Vega-lite for visualization, what say?

Vega should support the full Elasticsearch search DSL, so any query you can create using Console you can use within Vega.

Thats great! I think this should be rather straight forward, instead of creating an alias also.

Can you help me with this use case?

Suppose I have two visualization objects on my dashboard.
1 - Control
2 - Vega-lite

Control lets the user choose a process name and vega-lite displays the process name.

After the user selects Process A,

If the number of processes is 10001, given vega-lite takes 10000 records, is there is a way to achieve this?

Kindly let me know if I should post this query in a separate thread. Thanks!