Spark sql query for empty string

Elastic Stack Elasticsearch
1

I am using spark sql to query for an empty string . Documents are returned exactly when I use elasticsearch dsl as following:
user/_search
{
"query": {
"term": {
"type": ""
}
}
}
But in spark sql, I written a sql like "select * from user where type = '' ", It does not return any results.
how to query for empty string in spark sql ?

2

try setting the strict setting on the job settings to true. This will instruct the connector to use term queries instead of match queries in the DSL it generates. See the Spark SQL docs for more information about those settings.

3

thank you. I try to set the param "strict = true", but it does not work exactly. Then I set the param "double.filtering = false", It can work exactly. Why I must disable "double.filtering" when turning strict on?
The param "double.filtering" is really tricky.

4

This may be an issue within Spark itself if you have to disable double.filtering. Spark likes to double check that pushdown operations have actually filtered everything correctly, so it's possible that Spark is interpreting the filter statements differently than ES-Hadoop does in this case.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.