Elastic sink from Google Dataflow/Apache Beam

Elastic Stack Elasticsearch
1

Hi all,
Moving forward with using Elasticsearch, our next phase is moving from Spark jobs that writes into Elastic, to Apache Beam pipelines that will do the same.

While we have Apache Beam / Google Dataflow sinks for BigQuery, BigTable and more - I need to find a sink for Elastic.

Any change anyone has something ready, or some code examples ?
Anyone else need to achieve the same goal ?

Thanks,
Shushu

2

Hi Shushu,
Apache Beam doesn't currently include an Elasticsearch connector out-of-the-box. However, Beam has an extensible API for writing such connectors, and any user should be empowered to do so relatively quickly.

That said, we'd like to have an Elasticsearch connector in Beam to make it easier for everyone. I know some members of the Beam community have started working on this. Perhaps you can reach out to us at dev@beam.incubator.apache.org and jointly collaborate on this connector.

Thanks,
Davor

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.