I am looking for a solution to get all the records from the elasticsearch for visualization.
It seems only 10000 documents can be visualized at a time.
I went through elasticsearch documentation and found out that "scroll" would be an option. I could use this approach in python to loop through all the documents no matter how many documents exists in elasticsearch.
But i want to get all the documents for visualization in a scatter plot and i am trying to find a way to include the main search query (which would get me default 10000 docs) together with the scroll query (which would get me the all or remaining documents). In dev tools (console) of kibana, i have to use two separate queries but not sure about the way to use these queries in scatter plot or in any other visualization tools. It seems combining these two queries together wont work.
Hi there, you should be able to increase this limit by changing the index.max_result_window setting in Elasticsearch. See the docs on index settings for more info.
Technically you can bump this limit on the Elasticsearch side by adjusting the index.max_result_window. Just be aware there may be an associated performance cost.. That would really be the only option for increasing the number of documents returned to a Kibana visualization.
Outside of Kibana, as you said, the Scroll API is your best best for exceeding this limit.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.