Vega & response filtering (filter_path)

Elastic Stack Kibana
1

Hi,

I've been playing with Vega (love it!) and I have a use case that is generating a lot of unnecessary data that is slowing kibana/vega to a crawl. It's because I am doing a pipeline aggregation of an aggregation...

I want to find the sum of lots of things and then the average of all of those. But I only need to plot the final average so the returned aggregation of sums is not required in vega. This represents the majority of the data being processed and stored by vega.

Now if I was to do this with a rest/dsl call I'd use "filter_path" to get rid of the intermediate data results (e.g. How to filter out useless aggregation result when using pipeline aggregations).

Is something like this possible in vega or would it make a good feature request? I had a look at the transform filter but I'm not sure that actually drops data or helps reduce the initial load time.

Daire

2

@nyuriks thanks :slight_smile:

3

@Daire_Byrne would it be possible for you to do all of the processing on the server? Ideally, you should make ES produce just the data you need for drawing.

4

Well, I would be happy to do that but I'm not aware of a way using the current layout of my dataset without using a pipeline aggregation. Maybe others can suggest an alternative...

I needed to use a pipeline aggregation to group by term and then group by another term (and sum). Finally, I need the overall average of those.

The problem with the pipeline aggregation is that there doesn't seem to be a way to avoid returning the intermediate data other than by filtering (like the previous topic I linked). If you don't return this first aggregate data then the pipeline aggregate of the aggregate (the data I care about) also won't return any data either.

BTW really loving the Vega integration. It has opened up a lot of interesting possibilities that the standard kibana options just couldn't deal with.

Daire

5

Daire, as you considered using a REST access in your original post, have you maybe tried doing data acquisition via a URL? Mentioned by the intro video at this time. @timroes advises that you need to manually enable this option in the kibana.yml file (for security reasons).

6

Robert,

If I understand correctly, the main problem with grabbing the data directly via a URL is that I would lose the interaction with the Kibana search bar. So the Kibana dashboard queries will not be part of the URL.

Obviously one of the most appealing features of the vega kibana integration is this ability to have live interactions with the data.

I just can't seem to think of any other way to return the results I need without using a pipeline aggregation and all the intermediate data that entails.....

Daire

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.