Export From elasticsearch to S3


(praveen.vemuri) #1

Hi,

What is the best way to export raw data from elasticsearch to s3?

i read about elasticdump and there we have to download to local filesystem and then copy to s3?


(Wayne Taylor) #2

Have you tried using logstash and output to s3? https://www.elastic.co/guide/en/logstash/current/plugins-outputs-s3.html

You can use elasticsearch as your input source


(praveen.vemuri) #3

Hi Wayne, Thanks for your response..

I tried it once couple months back. but it is restarting randomly and starting the export from begining again. so i didn't try it now in 6.2.4. i will try it now and let you know.


(Wayne Taylor) #4

Praveen,

Do you have a sincedb configured and what version did you have that problem with? I recall having similar issue on restarts but an update solved it.

Wayne


(praveen.vemuri) #5

Hi Wayne, I just tried with logstash 6.2.4 and it restarted. i tried to configure sincedb_path but it says unknown setting. As elasticsearch depends on scroll id. is it configurable for elasticsearch input?

[LogStash::Runner] elasticsearch - Unknown setting 'sincedb_path' for elasticsearch


(Wayne Taylor) #6

Sorry - my bad. Since is for input not output.

Can you put in debug mode and provide output here? Also, this needs to move from Elasticsearch to Logstash forum. You'll get better responses


(system) #7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.