Elastic/Kibana data exportaion (over 1b hits)

Elastic Stack Elasticsearch
1

Hello.

I have the assignment to export specific client data from the elastic to any compatible format (csv, json, ndjson, etc.)

There is an option to generate a CSV Report from Kibana by a saved query.
image

However, this option isn't working for me for a few reasons:

It has stringent size limits - I tried to download 300k hits but got only 130k (because of the max size limit reached).

image

In fact, I have to download billions of rows (1.6 billion rows for the last 30 days). Using this method isn't a good way to save time.

Are there any other methods I can use?

I'm thinking about some tool that can reach the elastic database from the backend (I saw elasticdump, but don't know much about it).

I will be so glad for any tips or hints.

Thank you all!

2

Hi @yzaritskyi,

Welcome back. There is a 10MB limit by default for exporting from Kibana. It can be configured via the xpack.reporting.csv.maxSizeBytes setting in kibana.yml for your cluster as per this thread . But I would tread carefully there as it can impact performance if set too high.

This thread gives a couple of options that you could try, including:

  1. Writing a script using a language client and a CSV utility, such as Python. There is a StackOverflow thread here that shares that example.
  2. Using Logstash with an Elasticsearch filter processor and the CSV output plugin .
  3. Split your query into multiple to export the records across multiple CSV files.

Let us know if any of those options work for you.

1 Like
3

Thank you for the info. From what you suggested - I'm looking into logstash as the most automized option. However, I'd like to ask if you know anything elastic-dump tool. Is it authorized by elastic devs to use for data exportation?

Thanks!

4

Hi @yzaritskyi,

By elastic-dump tool do you mean this:

It's not a tool I've come across before, and it looks to be community maintained. It's definitely not an Elastic tool that we support.

It's really up to yourself whether you would like to use the tool, but I would read their instructions and specifically version caveats carefully.

Hope that helps!

1 Like