Hi! Sure. I do want to create index and type in advance however I want them to be created at runtime. so say I get stream of messages and apply some transformation which will result in a DataSet. Now, this result DataSet will have a schema right so I want to take this schema and create a index and type while mapping to ES equivalent types and start appending docs.
I can do this df.select(to_json(functions.struct(col("*"))).as("value")) and this gives me a json which I want to write it as a doc in Elastic search.
Does that make sense? How can I achieve it?