How to get documents more than 10000 in elasticsearch for visualization?

[Nirajan]

Hello,

I am looking for a solution to get all the records from the elasticsearch for visualization.
It seems only 10000 documents can be visualized at a time.

I went through elasticsearch documentation and found out that "scroll" would be an option. I could use this approach in python to loop through all the documents no matter how many documents exists in elasticsearch.

But i want to get all the documents for visualization in a scatter plot and i am trying to find a way to include the main search query (which would get me default 10000 docs) together with the scroll query (which would get me the all or remaining documents). In dev tools (console) of kibana, i have to use two separate queries but not sure about the way to use these queries in scatter plot or in any other visualization tools. It seems combining these two queries together wont work.

Any suggestions would be really appreciated..

Best Regards,
Nirajan

[cjcenizal]

Hi there, you should be able to increase this limit by changing the index.max_result_window setting in Elasticsearch. See the docs on index settings for more info.

[Aaron_Caldwell]

Hello,

I'll echo what CJ said.

Technically you can bump this limit on the Elasticsearch side by adjusting the index.max_result_window. Just be aware there may be an associated performance cost.. That would really be the only option for increasing the number of documents returned to a Kibana visualization.

Outside of Kibana, as you said, the Scroll API is your best best for exceeding this limit.

Regards,
Aaron

[Nirajan]

Hello Guys,

Thanks a lot for your suggestions.

I tried it and it works.

Have a good time.

Best Regards,

Nirajan