Using Elastic stack

Elastic Stack Logstash
1

Hi,
As a new user of Elastic, I'm trying to to establish an architecture for my needs. I'll describe my Dilemmas:
I'm using a service API(CA agile control) that holds thousands data records. In this API, it is possible to get maximum 2000 records at a time.
Due to Logstash limitation, it cannot indexing my records (e.g., read 1st 2000 records and than starts from record index 2001 for the next 2000 records and so on..). There is no straight forward iteration solution based on Logstash configuration file

I can create an intermediate toolkit that iterate over my service API, for each 2000 record by index, and send it to Logstash - and from there to Elasticsearch and Kibana.

My questions are(For the long term):

  • What are the benefits of using Elastic stack in this case?
  • according to above, is it necessary to use Logstash? is it better to set my toolkit straight forward to Elasticsearch?
  • I'll need to use Beats component to ship data from more than 1 data source. Can beats talk directly with Elasticsearch and skip Logstash? If not, what are the advantages?

I hope I made myself clear. Sorry for the length. I'm trying to set up my needs as clear as possible
Thanks

2

What are the benefits of using Elastic stack in this case?
according to above, is it necessary to use Logstash? is it better to set my toolkit straight forward to Elasticsearch?

It's never necessary to use Logstash but it's often convenient, but for some more specialized use cases it's inadequate.

I'll need to use Beats component to ship data from more than 1 data source. Can beats talk directly with Elasticsearch and skip Logstash? If not, what are the advantages?

Yes, it can.

3

Tnx @magnusbaeck
By saying Logstash it's often convenient, what do you mean? in which cases?

4

Do you want me enumerate all the cases where using Logstash is easier than writing a custom script?

5

No. off course not :slight_smile:
Any relevant links/tutorial that you can recommend? for this case

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.