When using inner_hits on nested query, we are getting an index_out_of_bounds_exception


(Phil Yardley) #1

Hi,

I'm pulling my hair out on this (and I don't have much left to pull out!)

We have a set of documents which are replicated to Elastic from Couchbase.

We're new to Elastic, so feel free to say "Why are you doing this, this is a better way" (so long as it's constructive :wink: )

in the document, we have a "holiday product", this product has a nested array of "prices", this price object contains fields such as number of passengers, date, promo code, price etc. For each holiday product, this list of prices can be very long as there can be hundreds/thousands of permutations.

When running a query against the data, we don't want to carry the 1000's of lines of data over the wire (can be >5MB) so are trying to use inner_hits to only return the rows that match. (approx 10KB)

we have a query such as:

{
"_source": false,
   "query": {
    "nested": {
      "path": "doc.products.calculatedPrices",
      "score_mode": "avg",
        "query": {
        "bool": {
          "must": [
            { "match": { "doc.products.calculatedPrices.promoCode": "DRH725C" } },
            { "match": { "doc.products.calculatedPrices.pax": 2 } },
            { "range": { "doc.products.calculatedPrices.now" : {"gte": 100, "lte":120} } }
           ]
        }    
	  },
	  "inner_hits": {}
    }
  }
}

this generates the error:

{
    "took": 41,
    "timed_out": false,
    "_shards": {
        "total": 3,
        "successful": 2,
        "failed": 1,
        "failures": [
            {
                "shard": 2,
                "index": "testavail5",
                "node": "AfXKKfqrSPqOrNO7mLreEQ",
                "reason": {
                    "type": "index_out_of_bounds_exception",
                    "reason": "Index: 7290, Size: 134"
                }
            }
        ]
    },
    "hits": {
        "total": 62,
        "max_score": 5.845086,
        "hits": []
    }
}

On this test server, we are running a single node with 3 shards. so not sure why one share reports as failed.

if we change the
inner_hits": {} to inner_hits": {"_source":false}

we get results but obviously don't get any useful information in the output!

Are we hitting a bug in ES 5.4 ? (or more likely) are we doing something stupid?

Thanks

Phil

(happy to post more snippets / logs etc as required - just didn't want to overload the post with too much info that might not be needed)


(Martijn Van Groningen) #2

@BLZB0B I think this is a bug too. Do you see any stacktrace with the Fetch phase failed in elasticsearch's log file?

If you don't see any stacktraces, can you then add the following to the log4j2.properties file?

logger.search.name = org.elasticsearch.search
logger.search.level = trace

This should make the stacktrace visible. If you like you can also open an issue on github, since this is a bug.


(Phil Yardley) #3

Hi Martijn,

we get the following:

[2017-06-20T11:46:58,972][TRACE][o.e.s.SearchService      ] [AfXKKfq] Fetch phase failed
java.lang.IndexOutOfBoundsException: Index: 7290, Size: 134
        at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[?:1.8.0_131]
        at java.util.ArrayList.get(ArrayList.java:429) ~[?:1.8.0_131]
        at org.elasticsearch.search.fetch.FetchPhase.createNestedSearchHit(FetchPhase.java:256) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:150) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.search.fetch.subphase.InnerHitsFetchSubPhase.hitExecute(InnerHitsFetchSubPhase.java:65) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:161) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:417) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.action.search.SearchTransportService$12.messageReceived(SearchTransportService.java:394) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.action.search.SearchTransportService$12.messageReceived(SearchTransportService.java:391) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:627) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.4.0.jar:5.4.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
[2017-06-20T11:46:58,973][DEBUG][o.e.a.s.TransportSearchAction] [AfXKKfq] [6] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [AfXKKfq][172.31.35.8:9300][indices:data/read/search[phase/fetch/id]]
Caused by: java.lang.IndexOutOfBoundsException: Index: 7290, Size: 134
        at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[?:1.8.0_131]
        at java.util.ArrayList.get(ArrayList.java:429) ~[?:1.8.0_131]
        at org.elasticsearch.search.fetch.FetchPhase.createNestedSearchHit(FetchPhase.java:256) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:150) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.search.fetch.subphase.InnerHitsFetchSubPhase.hitExecute(InnerHitsFetchSubPhase.java:65) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:161) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:417) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.action.search.SearchTransportService$12.messageReceived(SearchTransportService.java:394) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.action.search.SearchTransportService$12.messageReceived(SearchTransportService.java:391) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:627) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.4.0.jar:5.4.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

Thanks

Phil


(Martijn Van Groningen) #4

Thanks Phil, this is very helpful.


(Phil Yardley) #5

Hi Martijn,

Github bug raised as suggested : https://github.com/elastic/elasticsearch/issues/25315

are there any good work arounds that you could suggest? "all we need to do" is to reduce the inner results of the array before returning the data.

Thanks

Phil


(Martijn Van Groningen) #6

What you can try to do as a workaround is to use doc_values fields inside the inner hit definition:
https://www.elastic.co/guide/en/elasticsearch/reference/5.4/search-request-docvalue-fields.html

Most fields nowadays have doc values enabled by default (fields mapped as text don't have doc values), so you should be able to return most of your nested fields.


(Phil Yardley) #7

thanks - will take a look now.

Regards

Phil


(Phil Yardley) #8

Hi @mvg ,

Unfortunately, I don't think this is an option as we're running nested queries:

[nested] query does not support [docvalue_fields]

:frowning:

Regards

Phil


(Martijn Van Groningen) #9

Can you share the query you were trying to execute? Looks like docvalue_fields isn't specified in the inner_hits itself, but directly in the nested query.


(Phil Yardley) #10

@mvg Sorry, built a dodgy query...

corrected it but not quite getting the expected results:

{
"_source": false,
   "query": {
    "nested": {
      "path": "doc.products.calculatedPrices",
      "score_mode": "avg",
        "query": {
        "bool": {
          "must": [
            { "match": { "doc.products.calculatedPrices.promoCode": "DRH725C" } },
            { "match": { "doc.products.calculatedPrices.pax": 2 } },
            { "range": { "doc.products.calculatedPrices.now" : {"gte": 100, "lte":120} } }
           ]
        }
      
	  },
	  "inner_hits":{  // The actual list would be slightly longer but just these fields will be fine for the moment
                "stored_fields" : ["doc.products.calculatedPrices.was","doc.products.calculatedPrices.now"]
	  }
    }
  }
}

returns (snippet):

{
    "took": 106,
    "timed_out": false,
    "_shards": {
        "total": 3,
        "successful": 3,
        "failed": 0
    },
    "hits": {
        "total": 60,
        "max_score": 5.831132,
        "hits": [
            {
                "_index": "testavail5",
                "_type": "couchbaseDocument",
                "_id": "CLMOEA2",
                "_score": 5.831132,
                "inner_hits": {
                    "doc.products.calculatedPrices": {
                        "hits": {
                            "total": 42,
                            "max_score": 5.831132,
                            "hits": [
                                {
                                    "_nested": {
                                        "field": "doc.products.calculatedPrices",
                                        "offset": 11087
                                    },
                                    "_score": 5.831132
                                },
                                {
                                    "_nested": {
                                        "field": "doc.products.calculatedPrices",
                                        "offset": 10995
                                    },
                                    "_score": 5.831132
                                },
                                {
                                    "_nested": {
                                        "field": "doc.products.calculatedPrices",
                                        "offset": 10365
                                    },
                                    "_score": 5.831132
                                }
                            ]
                        }
                    }
                }
            }, <snip>

thanks

Phil


(Martijn Van Groningen) #11

That is because stored_fields is used instead of docvalue_fields, can you try this:

{
"_source": false,
   "query": {
    "nested": {
      "path": "doc.products.calculatedPrices",
      "score_mode": "avg",
        "query": {
        "bool": {
          "must": [
            { "match": { "doc.products.calculatedPrices.promoCode": "DRH725C" } },
            { "match": { "doc.products.calculatedPrices.pax": 2 } },
            { "range": { "doc.products.calculatedPrices.now" : {"gte": 100, "lte":120} } }
           ]
        }
      
	  },
	  "inner_hits":{  // The actual list would be slightly longer but just these fields will be fine for the moment
                "docvalue_fields" : ["doc.products.calculatedPrices.was","doc.products.calculatedPrices.now"]
	  }
    }
  }

Stored fields are disabled by default (except for _source field), so it is expected that you don't see any key value pairs.


(Phil Yardley) #12

@mvg

hi,

That takes us back to the "index_out_of_bounds_exception" again :frowning:

[2017-06-20T15:37:40,783][TRACE][o.e.s.SearchService      ] [AfXKKfq] Fetch phase failed
java.lang.IndexOutOfBoundsException: Index: 11087, Size: 77
        at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[?:1.8.0_131]
        at java.util.ArrayList.get(ArrayList.java:429) ~[?:1.8.0_131]
        at org.elasticsearch.search.fetch.FetchPhase.createNestedSearchHit(FetchPhase.java:256) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:150) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.search.fetch.subphase.InnerHitsFetchSubPhase.hitExecute(InnerHitsFetchSubPhase.java:65) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:161) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:417) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.action.search.SearchTransportService$12.messageReceived(SearchTransportService.java:394) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.action.search.SearchTransportService$12.messageReceived(SearchTransportService.java:391) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:627) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.4.0.jar:5.4.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
[2017-06-20T15:37:40,784][DEBUG][o.e.a.s.TransportSearchAction] [AfXKKfq] [1014] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [AfXKKfq][172.31.35.8:9300][indices:data/read/search[phase/fetch/id]]
Caused by: java.lang.IndexOutOfBoundsException: Index: 11087, Size: 77
        at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[?:1.8.0_131]
        at java.util.ArrayList.get(ArrayList.java:429) ~[?:1.8.0_131]
        at org.elasticsearch.search.fetch.FetchPhase.createNestedSearchHit(FetchPhase.java:256) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:150) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.search.fetch.subphase.InnerHitsFetchSubPhase.hitExecute(InnerHitsFetchSubPhase.java:65) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:161) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:417) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.action.search.SearchTransportService$12.messageReceived(SearchTransportService.java:394) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.action.search.SearchTransportService$12.messageReceived(SearchTransportService.java:391) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:627) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.4.0.jar:5.4.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

Appreciate your patience and help on this...


(Martijn Van Groningen) #13

Oops, the nested source fetching needs to be disabled (by adding "_source": false to inner_hits):

{
"_source": false,
   "query": {
    "nested": {
      "path": "doc.products.calculatedPrices",
      "score_mode": "avg",
        "query": {
        "bool": {
          "must": [
            { "match": { "doc.products.calculatedPrices.promoCode": "DRH725C" } },
            { "match": { "doc.products.calculatedPrices.pax": 2 } },
            { "range": { "doc.products.calculatedPrices.now" : {"gte": 100, "lte":120} } }
           ]
        }
      
	  },
	  "inner_hits":{  // The actual list would be slightly longer but just these fields will be fine for the moment
                "docvalue_fields" : ["doc.products.calculatedPrices.was","doc.products.calculatedPrices.now"],
                "_source": false
	  }
    }
  }

(Phil Yardley) #14

@mvg

haha... Yes :smile:

getting results now!

I'll expand the above example to include additional fields so the results make more sense - I'll need to update our mapping to include keywords on some of the text fields

"Fielddata is disabled on text fields by default. Set fielddata=true on [doc.products.calculatedPrices.promoCode] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."

but that should be simple enough (he says)

Thanks again

Phil


(Phil Yardley) #15

Hi Martijn,

thanks for your help so far - the last issue (for the moment) is as follows.

You've probably seen in our data, we have a doc, which has nested products, which have nested calculatedPrices.

when we run a query, the inner hits for calculatedPrices need to be at the product level not at the doc level eg:

  "hits": {
        "total": 241,
        "max_score": 4.94516,
        "hits": [
            {
                "_index": "testavail5",
                "_type": "couchbaseDocument",
                "_id": "COHARA2",
                "_score": 4.94516,
                "_source": {
                    "doc": {
                        "destinationCode": "HAR",
                        "productSku": "Harrogate, St George Hotel",
                        "productType": "SelfDrive"
                    }
                },
                "inner_hits": {
                    "doc.products.calculatedPrices": {
                        "hits": {
                            "total": 198,
                            "max_score": 4.94516,
                            "hits": [
                                {
                                    "_nested": {
                                        "field": "doc.products",
                                        "offset": 32,
                                        "_nested": {
                                            "field": "calculatedPrices",
                                            "offset": 101
                                        }
                                    },
                                    <blah blah>

is incorrect... as the inner hits are specific to a particular product within the doc.

it needs to be something like but not exaclty :

  "hits": {
        "total": 241,
        "max_score": 4.94516,
        "hits": [
            {
                "_index": "testavail5",
                "_type": "couchbaseDocument",
                "_id": "COHARA2",
                "_score": 4.94516,
                "_source": {
                    "doc": {
                        "destinationCode": "HAR",
                        "productSku": "Harrogate, St George Hotel",
                        "productType": "SelfDrive",
						"inner_hits": {
						"doc.products.calculatedPrices": {
							"hits": {
								"total": 198,
								"max_score": 4.94516,
								"hits": [
									{
										"_nested": {
											"field": "doc.products",
											"offset": 32,
											"_nested": {
												"field": "calculatedPrices",
												"offset": 101
											}
										},
										<blah blah>

is this even possible? any advice / pointers / documentation to read would be much appreciated.

Regards

Phil


(Martijn Van Groningen) #16

Hi Phil,

The reason the inner hit is directly returned doc.products.calculatedPrices level is because doc and products fields are object fields.

I think the the structure of the hits will be better if you make products field nested too. By doing that you will need to use the nested query more in order to query the product properties.

Your query will then look like this:

{
  "_source": false,
  "query": {
    "nested": {
      "path": "doc.products",
      "score_mode": "avg",
      "inner_hits": {
        "_source": false
      },
      "query": {
        "nested": {
          "path": "doc.products.calculatedPrices",
          "score_mode": "avg",
          "query": {
            "bool": {
              "must": [
                {
                  "match": {
                    "doc.products.calculatedPrices.promoCode": "DRH725C"
                  }
                },
                {
                  "match": {
                    "doc.products.calculatedPrices.pax": 2
                  }
                },
                {
                  "range": {
                    "doc.products.calculatedPrices.now": {
                      "gte": 100,
                      "lte": 120
                    }
                  }
                }
              ]
            }
          },
          "inner_hits": {
            "docvalue_fields": [
              "doc.products.calculatedPrices.was",
              "doc.products.calculatedPrices.now"
            ],
            "_source": false
          }
        }
      }
    }
  }
}

I hope this helps.

Martijn


(Phil Yardley) #17

Many thanks,

I will give this a try shortly. We have already set products as nested in the mapping based on your previous suggestions so we should be in a good place to try the above.

Many thanks again..

Phil


(Phil Yardley) #18

Hi Martijn,

We're still building this :frowning: what we've noticed is that as the amount of nested data increases, there is a huge slowdown in the results. I'm guessing this is to be expected to a certain extent - but it seems to be an exponential slow down.

Based on what you know of our data - would reshaping the data be beneficial? would reducing from 2 levels of nesting to one be beneficial? would flattening all of the data from 6000 "rows" to 20m "rows" have it's own performance challenges ?

thanks

Phil


(Martijn Van Groningen) #19

Hi Phil,

I think flatting the data will for sure have a positive impact on the performance of your queries. For example I think that the doc nested field doesn't add a whole lot of meaning, so I would remove that nested level. You can also duplicate the product data in each price nested object, but that is going to increase the amount of document like you said. ES should be able to handle this fine, but it does mean that your application needs to handle the search responses of certain queries differently (data isn't structured the same any more).

Maybe before going down this path, it would be good to know what is exactly slow. If you run your queries in a continuous loop and at the same time capture hot threads then I'll be happy to look at it.

Martijn


(system) #20

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.