Read.field.include with nested field doesnot really work

Elastic Stack Elasticsearch
1 
CREATE TEMPORARY TABLE trips USING org.elasticsearch.spark.sql OPTIONS (
    nodes "....", port "80", 
    resource "events-2018.09.03/events", 
    read.field.include "client.browser.os"
)

select client.browser.os from trips limit 10

This is working fine, but, the generated elasticsearch query is:

{
	"slice": {
		"id": 20,
		"max": 21
	},
	"query": {
		"match_all": {}
	},
	"_source": ["client"]
}

Why _source is not : client.browser.os ?

2

It seems that Spark is not giving the nested field when calling the plugin "buildScan(requiredColumn ...)" method, so the plugin doesnt know.

So this means push down is not working with nested fields?

I see MongoDB has fixed this error: https://github.com/Stratio/Spark-MongoDB/blob/master/spark-mongodb/src/main/scala/com/stratio/datasource/mongodb/MongodbRelation.scala#L149

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.