Spark Structured Streaming using ES as sink

Elastic Stack Elasticsearch
1

Hello everybody,:wave:

I know the question has already been asked in the same terms but I think it deserves an update. Indeed, the new Spark 2.2 release now provide the structured streams in GA status (general availability), meaning it's no longer experimental.
So does ES support the structured stream now and if not, when and which release?
If not supported, is there a workaround meanwhile?

Thanks.

2

@Thomas_Rouch We released experimental support for Spark Structured Streaming in our ES-Hadoop 6.0.0-alpha2 release. Do note that the experimental support that was released will only work with Spark 2.1.0 (due to the fact that it was developed while the feature was still experimental). In the next early access release for ES-Hadoop 6.0, we will have support for Spark Structured Streaming as long as you are running Spark 2.2.0 and up (The GA release for Structured Streaming).

You can also take a look at the github issue for Structured Streaming support.

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.