ScalaMapFieldExtractor Cannot extract value from entity

Elastic Stack Elasticsearch
1

Hello there.
I have an RDD[SubscriptionInfo] wher SubscriptionInfo is a defined as a case class .

I'm trying to save the rdd to ES and I want to specify the _id to use, so I'm using something like this:

rdd.saveToEs("spark/docs", Map("es.mapping.id" -> "sub_id"))

but I get an error:

Caused by: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: [ScalaMapFieldExtractor for field [[sub_id]]] cannot extract value from entity [class java.lang.String] | instance [Some(SubscriptionInfo(ea9db3d23862862e68886de79bd2d489_4307394759,None,Some(List(Billing(Sun May 07 00:03:06 UTC 2017,2.0,None))),None,None,2.0,EmptyLTV()))]

Which is the correct way to specify the mapping id in this case?

2

It seams that the solution is to use and additional map stage, as suggested in https://discuss.elastic.co/t/cannot-extract-value-from-entity/66556

rdd.map({case Some(subscriptionInfo) => subscriptionInfo}).saveToEs("spark/docs", Map("es.mapping.id" -> "sub_id"))

3

My advice would be to avoid using Optional classes with ES-Hadoop since most of the logic is null aware already.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.