Filtering by Number of Nested Objects

[krmathieu (Kevin M.)]

I am trying to find all documents in an index where at least one nested object (segments.entities) has more than one value and here is the relevant snippet from the mapping:

..........
"segments" : {
            "type" : "nested",
            "include_in_parent" : true,
            "properties" : {
              "anchor" : {
                "type" : "nested",
                "include_in_parent" : true,
                "properties" : {
                  "section" : {
                    "type" : "string"
                  },
                  "start" : {
                    "type" : "long"
                  },
                  "stop" : {
                    "type" : "long"
                  },
                  "text" : {
                    "type" : "string",
                    "index" : "not_analyzed"
                  }
                }
              },
              "entities" : {
                "type" : "nested",
                "include_in_parent" : true,
                "properties" : {
                  "anchorText" : {
                    "type" : "string",
                    "index" : "not_analyzed"
                  },.........

And here is my query:

{
	"size": 100,
	"fields": ["document.content", "segments.entities.anchorText"],
    	"filter": {
		"script": {
			"script": "doc['segments.entities'].values.length > 1"
		}
	}
}

Running this query results in an error stating "No field found for [segments.entities] in mapping with types [doc]". If I remove the filter bit from the query above I get the expected results back so I know that the segments.entities path is valid. The field segments.entities should be an array and I am trying to find all documents where at least one segments.entities array has more than one element. Does anybody see what I am doing wrong? Any help would be appreciated. Thanks.

Kevin

[GyllingSW]

Try to see if this issue would do the trick:

Scripted filter on nested field

I have to filter documents based on scripted value of nested element. My data structure looks like this: { "product-group-name" : "group1" "products": [ { "name" : "product1" "price" : 10 },{ "name" : "product2" "price" : 20 } ] } "products" is nested field. Now I need to find product groups (top level documents) where one of the nested "price" fields satisfy condition I define with the script (I have complex script expression that can not be replaced with simple nested q…

[krmathieu (Kevin M.)]

I just tried updating my query and still see the following error:

No field found for [segments.entities] in mapping with types [doc]

Any idea what this means? Also, below is my updated query. Any additional thoughts that you may have would be appreciated. Thanks.

Kevin

{
"size": 100,
"filter": {
"nested": {
"path": "segments.entities",
"filter": {
"script": {
"script": "if (doc['segments.entities'].value > 1) {return (true)}"
}
}
}
}
}

[GyllingSW]

Change the Path value to:

"path" : "segments"

The path value is to the parent level compared to the actual query. If this doesn't help please post a snippet of your actual mapping

[krmathieu (Kevin M.)]

I changed my path to just segments and am still seeing the same issue when using Advanced REST client and the relevant mapping snippet is below. I did not create the mapping so I am not sure why the include_in_parent bit is used for segments when it is on the root level of the document. Any ideas why I am still seeing the problem?

Kevin

..........
"segments" : {
            "type" : "nested",
            "include_in_parent" : true,
            "properties" : {
              "anchor" : {
                "type" : "nested",
                "include_in_parent" : true,
                "properties" : {
                  "section" : {
                    "type" : "string"
                  },
                  "start" : {
                    "type" : "long"
                  },
                  "stop" : {
                    "type" : "long"
                  },
                  "text" : {
                    "type" : "string",
                    "index" : "not_analyzed"
                  }
                }
              },
              "entities" : {
                "type" : "nested",
                "include_in_parent" : true,
                "properties" : {
                  "anchorText" : {
                    "type" : "string",
                    "index" : "not_analyzed"
                  },

..........

[krmathieu (Kevin M.)]

For giggles I created a Bash script to use curl to execute the query from a Linux box and see this error vice the one I am seeing when using Advanced REST client:

"type" : "missing_property_exception",
"reason" : "No such property: segments for class: 88a4e7b9685857189da450004518aa361a991a47"

Not sure if the info here is related to these particular errors or not:

https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting-security.html#modules-scripting-security

Thoughts???

[krmathieu (Kevin M.)]

Actually, the issue with the Bash script was unescaped quotes. Once I fixed that I get the same error as I see in Advanced REST client. Sorry for the confusion.

Kevin

[krmathieu (Kevin M.)]

Update to this issue. Instead of using doc I tried using the following query against _source:

{
	"size": 100,
	"fields": ["document.content", "segments.entities.anchorText"],
    	"filter": {
		"script": {
			"script": "_source.segments.entities.values.flatten().length > 1"
		}
	}
}

Now the error I am seeing is this even though I am "flattening" the array:

Cannot compare java.util.ArrayList with value '[]' and java.lang.Integer with value '1'

I also tried this query and it just hangs, but does not throw any errors:

{
	"size": 100,
	"fields": ["document.content", "segments.entities.anchorText"],
    	"filter": {
		"script": {
			"script": "_source.segments.entities.size() > 1"
		}
	}
}

Any ideas on what is going on here? Any help getting this sorted out would be appreciated. Thanks.

Kevin