EsHadoopRemoteException: illegal_argument_exception: only write ops with an op_type of create are allowed in data streams?

[Zephery_Wen]

I want to use Spark to save data to a data stream.

JavaEsSparkSQL.saveToEs(result, dataStream, map);

It seems like 'saveToEs' only allows index.

EsHadoopRemoteException: illegal_argument_exception: only write ops with an op_type of create are allowed in data streams?

How to solve it?

Thanks!

[Zephery_Wen]

Anyone can help?

[Keith_Massey]

Hi @Zephery_Wen. I think that you just need to set the hadoop configuration property es.write.operation to create. That is set to index by default. Data streams require this to be set to create since they are append only. See Index API | Elasticsearch Guide [8.9] | Elastic for additional information about this.

[Zephery_Wen]

Hi @Keith_Massey . I'm sorry for not describing my problem clearly. I want to merge or insert my data.

known as merge or insert if the data does not exist, updates if the data exists (based on its id).

So I have to set the hadoop configuration property es.write.operation to upsert . Seems like JavaEsSparkSQL.saveToEs not support it?

[Keith_Massey]

I don't think that that is a limitation of JavaEsSparkSQL.saveToEs -- it's a limitation of data streams. For example, I think even just hitting the Elasticsearch API directly you cannot upsert a document in a data stream.