Data ingest using curl

Elastic Stack Beats
1

Is there a way to pull logs from an external system into Elastic through curl command? If so, what are the recommendations on it's architecture.

Here is what I have in mind:

Setup a shell script to run curl command -> Dump data in to a local file -> Filebeat -> Logstash pipeline -> ElasticSearch (v7.x)

Just wondering if there are other modules that we can leverage within logstash/beats. I couldn't find any but want to be sure.

Thanks!

2

Metricbeat can call a HTTP endpoint - https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-http.html. That might do what you want.

3

We are intending to pull the application logs from the remote end point, not necessarily system metrics. Can filebeat do the same instead so we can pull remote logs and ingest into elastic?

4

No it cannot.