Write RDD/DF to Index with no type

Elastic Stack Elasticsearch
1

EDIT: Whoops, did not realize the single type indexes were ES 5.6!

Hi there,

We're trying to write an RDD (with Spark Streaming) to Elasticsearch with just an index and no type, however we keep getting:

org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: invalid pattern given my_index/

However, when I add a type, it works fine. I am calling saveToEs like this:

EsSparkStreaming.saveToEs(directStream, "my_index/my_type", Map("es.nodes"->"localhost", "es.index.auto.create" -> "true"))

From what I can remember, regular Spark batch jobs can do just an index but Streaming doesn't seem to support that. This is my dependency in my POM in case it matters:

<dependency>
    <groupId>org.elasticsearch</groupId>
    <artifactId>elasticsearch-spark-20_2.11</artifactId>
    <version>5.5.0</version>
</dependency>

Thanks in advance!

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.