Issue with writing Spark dataset to Elastic


I'm trying to write a Spark dataset to es but I get following error while executing following piece of code -

I'm also setting following options while trying to write the dataset

        addOption(Constants.ELASTICSEARCH_PORT, "9200");

Error encountered
org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: resource must be specified for Elasticsearch resources.

How do I know what else is required for it to write the data / to specify the resource ?

The documentation for ES-Hadoop Spark is available on our documentation.

In particular, for Spark SQL Streaming, you can refer to this documentation section.

Options should be passed es.resource with value index/type (e.g. itemsindex/_doc).
Or you can specify it in the save().

Also ensure Constants.ELASTICSEARCH_HOST is actually es.nodes and Constants.ELASTICSEARCH_PORT is actually es.port.

All the options can be found here.

Thank you !

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.