Design/approach to build a bridge between elasticsearch and RestAPIs

Elastic Stack Elasticsearch
1

Elastic Team,

I need to push bulk data into elasticsearch instance. There are REST API exposed by the client and we need to call those Rest endpoints (GET call) , we should get JSON response from REST APIs. These JSONs needs to be push at elasticsearch. Till than it looks quite simple.

As we are interacting with rest api, we need to fetch the latest data almost everyday and then need to push the latest JSON to ES (everyday). Based on my research, I've discovered 2 approaches. Can anyone share his/her experience / knowledge and suggest anything.

  1. Use Logstash http-poller, provide list of REST endpoints which needs to be called repetitively and send the out to the ES instance.
  2. Use RabbitMQ and apply multiple consumer to push the bulk data to ES instance e.g.

Rest API => Message producer => RabbitMQ => Multiple consumer => ES

Since we need a scalable solution which can support a huge data on daily basis. Please share your expert views.

Thanks
Deepak

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.