Forward Logs from Elasticsearch to external destination application

[vishakh]

I'm looking to forward already stored logs from Elasticsearch to an external application-source.

We're quite aware that we could leverage Logstash to distribute/split the new future-incoming logs between Elasticsearch and other external destination BUT we would like to forward our already existing logs as-well.

[vishakh]

Can anyone please suggest

[stephenb (Stephen Brown)]

Elasticsearch does not push data. It has a REST endpoint (pull).

You will need to use an additional tool take data from elasticsearch and push it to another destination.

Example, You can use Logstash with elasticsearch as an input and send data / documents to some other destination system.

There are many batch and streaming ETL tools that can use elasticsearch as a source.

[vishakh]

While waiting for the response, I did research and go through using Logstash & API's to export logs from Elasticsearch.

As per your response "There are many batch and streaming ETL tools that can use elasticsearch as a source."
@stephenb Can you please name few ETL tools that can help me achieve this objective? (Are you referring to queuing tools?)

[stephenb (Stephen Brown)]

Examples I have not looked close at each lately

Logstash
Stream Sets
Talend
Informatica
Elasticsearch has a JDBC connector so anything that reads JDBC.

[vishakh]

Thanks for the response @stephenb
Considering that we would export the logs from Elasticsearch by any above means, doesn't this impose a overhead on the ELK-Stack? as this would be querying huge amount of data across all the indices

[stephenb (Stephen Brown)]

Well sure,

Like any datastore, reads / writes require some level of compute / IO resources nothing is free

That is why often teams with this use case often split the ingest feed to elasticsearch to both elasticsearch and the other destination at ingest time.

[vishakh]

Thank you @stephenb for your response/feedback.