How to write rows of a Streaming DataFrame to multiple ElasticSearch indices?

Elastic Stack Elasticsearch
1

How to write rows of a Streaming DataFrame to multiple ElasticSearch indices? For example I want to write row1 of a dataframe to index1, row2 of a dataframe to index2 , row3 of a dataframe to index3 and so on ...I have about 15 different indices so it would a bit painful and not a very clean code If were to filter and call writeStream() 15 times. so I am wondering if there is any easier way like the kafka dataframe below?

If I take Kafka data frame example it has the following schema by default.

root
 |-- key: binary (nullable = true)
 |-- value: binary (nullable = true)
 |-- topic: string (nullable = true)
 |-- partition: integer (nullable = true)
 |-- offset: long (nullable = true)
 |-- timestamp: timestamp (nullable = true)
 |-- timestampType: integer (nullable = true)

so I can have a topic field for every row such that the kafka connector takes care of write rows to appropriate topics.

2

You could try using the multi-resource index writing feature described in the docs

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.