As a new user of Elastic, I'm trying to to establish an architecture for my needs. I'll describe my Dilemmas:
I'm using a service API(CA agile control) that holds thousands data records. In this API, it is possible to get maximum 2000 records at a time.
Due to Logstash limitation, it cannot indexing my records (e.g., read 1st 2000 records and than starts from record index 2001 for the next 2000 records and so on..). There is no straight forward iteration solution based on Logstash configuration file
I can create an intermediate toolkit that iterate over my service API, for each 2000 record by index, and send it to Logstash - and from there to Elasticsearch and Kibana.
My questions are(For the long term):
- What are the benefits of using Elastic stack in this case?
- according to above, is it necessary to use Logstash? is it better to set my toolkit straight forward to Elasticsearch?
- I'll need to use Beats component to ship data from more than 1 data source. Can beats talk directly with Elasticsearch and skip Logstash? If not, what are the advantages?
I hope I made myself clear. Sorry for the length. I'm trying to set up my needs as clear as possible