Export From elasticsearch to S3

Elastic Stack Logstash
1

Hi,

What is the best way to export raw data from elasticsearch to s3?

i read about elasticdump and there we have to download to local filesystem and then copy to s3?

2

Have you tried using logstash and output to s3? https://www.elastic.co/guide/en/logstash/current/plugins-outputs-s3.html

You can use elasticsearch as your input source

3

Hi Wayne, Thanks for your response..

I tried it once couple months back. but it is restarting randomly and starting the export from begining again. so i didn't try it now in 6.2.4. i will try it now and let you know.

4

Praveen,

Do you have a sincedb configured and what version did you have that problem with? I recall having similar issue on restarts but an update solved it.

Wayne

5

Hi Wayne, I just tried with logstash 6.2.4 and it restarted. i tried to configure sincedb_path but it says unknown setting. As elasticsearch depends on scroll id. is it configurable for elasticsearch input?

[LogStash::Runner] elasticsearch - Unknown setting 'sincedb_path' for elasticsearch

6

Sorry - my bad. Since is for input not output.

Can you put in debug mode and provide output here? Also, this needs to move from Elasticsearch to Logstash forum. You'll get better responses