Spark-ES saveToES without type

Elastic Stack Elasticsearch
1

Hi everyone,

We're trying to ingest a DataFrame to Elasticsearch using the Elasticsearch Spark library. In Elasticsearch 6.x, types are now gone. So, to make our data ingestion more compliant, we've been trying to ingest our data without an index type.

We're calling the saveToES method like this:

dataFrame.saveToEs("indexname", config.toMap)

Then we get the following error when this method is called:

Caused by: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: No type found; Types are required when writing. Expected [index]/[type], but got [indexname]

We're using the following dependencies in SBT:

"org.elasticsearch" % "elasticsearch-spark-20_2.11" % "6.2.2" exclude("org.scalatest", "scalatest_2.11"),
"org.elasticsearch.client" % "x-pack-transport" % "6.2.2",
"org.elasticsearch.client" % "transport" % "6.2.2"

We're also using Elasticsearch 6.2.2.

2

Until types are fully removed (in Elasticsearch 7.0), all indices still have one type - either specified by the user, either a default one (doc).
You can double check this by looking at your mappings.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.