I am trying to read a mutli-level nested field "affiliations.org.orgName" using spark connector in following index
{"userId": 402885,
"userUuid": "F2896A1F1A7D411A69",
"firstName": "TRISHA",
"lastName": "WEEMS",
"age": 42,
"gender": "FEMALE",
"address": {
"countryId": 232,
"country": "US",
"subdivisionId": 4074,
"subdivision": "TN",
"postalCode": null,
"city": null
},
"categories": [
66,
68,
75,
91,
92,
113,
120,
121,
122,
158
],
"affiliations": [
{
"org": {
"orgName": "Dick's Sporting Goods",
"categories": [
65,
68,
85
],
},
"team": {
"teamName": "Dick's Sporting Goods Retail Employees",
},
"expirationDate": null
}
],
}
using following
val myDF = spark.read.format("org.elasticsearch.spark.sql")
.schema(user_schema)
.option("es.nodes", "localhost")
.option("es.port", "9200")
.option("es.field.read.empty.as.null","true")
.option("es.index.read.missing.as.empty","true")
.option("es.mapping.include", "categories, affiliations.*")
.option("es.read.field.include", "userId, userUuid, age, firstName, categories, lastName, affiliations.org.orgName")
.option("es.read.field.exclude", "affiliations.expirationDate, affiliations.team, affiliations.location")
.option("es.read.field.as.array.include", "affiliations") //not working
.load("dev21-targeting-users-1/user")
But I keep getting null for the field of org.orgName in myDF
+--------------------+-----------+
| userUuid|org.orgName|
+--------------------+-----------+
|F2896A1F1A7D411C9...| null|
|900FF5FE93D749CA9...| null|
+--------------------+-----------+
My schema is
val user_schema = StructType(
StructField("userUuid", StringType) ::
StructField("org.orgName", ArrayType(StringType),true) ::Nil)
Note : When I query ES using postman , I get the correct result.