What is the best way to insert/update billion data to ES from API?

(Gobinda Nandi) #1

I am getting millions of data from Google API, Now I want to store it in ES. I know the technique to save data to ES using Logsatsh from JSON file. So should I create a JSON file and then try to save it. Or there is a better and sophisticated way like CURL?

Thanks in advance :slight_smile:

(Nik Everett) #2

You might want to write an application that reads from the API and uses Elasticsearch's _bulk API. You'll want some way to resume where you left of and, hopefully, some way to identify the documents that you wrote so if you accidentally get two copies of the document from a funky resume you'll just overwrite the first copy with the second.

(Gobinda Nandi) #3

thanks @nik9000 for replying .. I will follow your advice. Peace !!! :slight_smile:

(Gobinda Nandi) #4

what if I am using logstash??

(Nik Everett) #5

To be honest I don't know much about logstash. I'm sure it'd work but it isn't a tool I have much experience with. If you have specific questions I'd ask them in https://discuss.elastic.co/c/logstash