Increase Top values aggregation limit above 1000

[lerignoux]

Hello

I am trying to generate a graph with a lot of records (~10k) and get the top values in an horizontal bar chart.
It works well but the Top Values: Number of values is limited to 1000 entries meaning I cant see all my data.

I checked the doc, advanced settings and tried to search if this was a parameter of my shard/index but could not find how to increase this limit.
Would anybody know how to do it ?

PS: I guess this limit is here for performances reasons.

[Marco_Liberati]

Hi @lerignoux ,

welcome to the kibana community.
As you already suggested the limit for 1000 values is mostly due to performance reasons, but also on the visualization side, having already 1000 values on a screen 4000px wide ( > 4k ) means that there are no more than 4 pixels for each value bar.

Do you need to visualize all the values in your dataset? You have a particular use case for that?

[lerignoux]

Hello

Thank you for your help.

Actually I will not display all the values.
I am setting up an horizontal bar chart so records are aggregated together according to this slice top values. The problem is that because of this 1000 limit I cannot fetch all the records and some rows are not showing up:

For instance in a worse case if you have 1000 records A and 1 records B (that arrive last) you could end up with a chart showing only the row for A records and think B record does not exists.

Or maybe there is something else I didn't setup properly ?

[Marco_Liberati]

I think there's a misunderstanding.
The Number of values parameter sets a limit of the number of values, not records/documents fetched from the dataview: say you want to visualize the full dataview, and you visualize a field with only 3 values (i.e. success,warning,error) within 1 million documents, then any value above 3 for Number of values should work.

In your pic looks like there are less than 100 values in your documents for that field: perhaps the time range is too short? Or perhaps you have some filters enabled?

[lerignoux]

Hello

Thanks for the info, I currently have no other filter than the date range one.
Adding a filter on the right entries I do see them appear in the chart. So time range is correcty too.

I lowered the value and had more rows than my limit. Though, changing it does alter the aggregation limit sent to Elasticsearch:

I'm not so clear of what this aggregation limit means, Ill try to check the doc.

• checking ES doc it seems this parameter can go up to search.max_buckets so should be large.
  Though it should indeed return up to 1000 props. Though according to the doc, results can be indeed be missing but having only 1 shard it should not from what I understand.

I may still need to increase the max value but don't know how.

[lerignoux]

Ok I think I found my issue ... All the rows up to 1000 are indeed found and displayed if the chart is big enough vertically.
It was too shrink-ed and thus hiding results that could not legibly be displayed.

Thanks a lot @Marco_Liberati for helping me understand how all this worked.

Have a nice weekend