Sorry, its been a while but I kept this in the back of my mind and recently got another idea. Its a bit "hacky" and not nearly close to ready, but I though I might as well share it because the direction should be clear. The idea is to:
- store a document per Image
- include one multi-valued field with the "detected_entities_ids" (maybe some space efficient short version)
- alongside the other image properties and the above field, store a simple inner object under each detected object id that includes the bbox information.
- filter for images containing only object "a" and "b" (or more) in filter match clauses on the "detected_entities_ids" field
- add another script filter that accesses the bbox information for each required object, using the object id in the path to lookup the corner values.
Here is a simplified example of what I mean. The query should return document 1. The script models some kind of "a left of b" relationship, meaning that the left+width value of the "a" object must be smaller than the left value of the "b" object.
Caveats:
- I don't know how performant this is on larger datasets, but hopefully the first filter on the "detected_objects" field narrows the search space down enough for the script to not have to iterate over too many docs.
- real-world locality relationships might be harder to express, but it should be doable
- you need the image ids when you generate the query (maybe the script is parametrizable though) in order to use the right "lookup" path for the inner objects
There are a lot of edge cases to consider, e.g. this simple script breaks badly if any of the docs matching the first two filters doesn't contain either of the inner objects that match the "detected_object" fields. But it should be possible to manage those cases somehow.
Hope this gives you some direction to play with, I'm suprised this works somehow after all and is suprisingly short. Maybe I missed an important constraint, but there you go 
PUT test/_doc/1
{
"detected_objects" : [ "a", "b" ],
"a" : {
"Height": 0.05,
"Left": 0.4,
"Top": 0.6,
"Width" : 0.1
},
"b" : {
"Height": 0.05,
"Left": 0.6,
"Top": 0.6,
"Width" : 0.1
}
}
PUT test/_doc/2
{
"detected_objects" : [ "a", "b" ],
"a" : {
"Height": 0.05,
"Left": 0.6,
"Top": 0.6,
"Width" : 0.1
},
"b" : {
"Height": 0.05,
"Left": 0.4,
"Top": 0.6,
"Width" : 0.1
}
}
PUT test/_doc/3
{
"detected_objects" : [ "a", "c" ],
"a" : {
"Height": 0.05,
"Left": 0.4,
"Top": 0.6,
"Width" : 0.1
},
"c" : {
"Height": 0.05,
"Left": 0.6,
"Top": 0.6,
"Width" : 0.1
}
}
PUT test/_doc/4
{
"detected_objects" : [ "a", "b" ],
"a" : {
"Height": 0.05,
"Left": 0.4,
"Top": 0.6,
"Width" : 0.1
},
"b" : {
"Height": 0.05,
"Left": 0.5, <- This shouldn't match currently because the value is right on a.Left + a.Width, but you can play with it by "moving" it
"Top": 0.6,
"Width" : 0.1
}
}
GET /test/_search
{
"query": {
"bool": {
"filter": [
{
"match": {
"detected_objects": "a"
}
},
{
"match": {
"detected_objects": "b"
}
},
{
"script": {
"script": {
"source": """
return doc['a.Left'].value + doc['a.Width'].value < doc['b.Left'].value;
""",
"lang": "painless"
}
}
}
]
}
}
}