What is the correct approach to export xx mil of records from elastic using kibana

#1

For exporting huge amount of data it is recommended to use scroll api , but users of Kibana wants to export milions of record using without this coding. Shall I increase the limits or is there a better approach. Desired output is csv type of file and the query is constructed using Kibana.
Can you recommend please the best way how to handle this ?
Is there a tool in Kibana using scroll api?

Thank you

#2

for instance I am thinking : if I could have query in kibana and transfer it into some script/tool and execute it in order to generate desired csv file.

use case is:
users ara analysisng the data, and when they find the key of the problem they need to export all data with given key

#3

Hello Petr,

We have csv export from discover and dashboards. But two caveats:

  1. You need to have the default distribution of elasticsearch with basic license (its free)
  2. You will definitely hit size limits at one point.

What I am confused about is millions of documents? Usually if you query in Kibana the results will be based on your query. Do you anticipate your query results over millions of documents?

Thanks,
Bhavya

1 Like
#4

Thank you Bhavya
Yes the we create the query search in Kibana, we add columnt to prepare the shape of the CSV and we need to export the events.
Our cluster is 11 data nodes and 2 kibana nodes.
The users of our elasticsearch have reasons to export milions of documents. I gave them the scroll api reference, but as I wrote no everyone is skilled in coding the query.

#5

@Bargs how do we help this user?

Thanks,
Bhavya

1 Like