# When using inner\_hits on nested query, we are getting an index\_out\_of\_bounds\_exception

**URL:** <https://discuss.elastic.co/t/when-using-inner-hits-on-nested-query-we-are-getting-an-index-out-of-bounds-exception/89968>\
**Category:** Elasticsearch\
**Created:** [June 19, 2017, 4:11pm UTC](https://discuss.elastic.co/t/when-using-inner-hits-on-nested-query-we-are-getting-an-index-out-of-bounds-exception/89968 "2017-06-19T16:11:57Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![BLZB0B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blzb0b/32/24714_2.png) [@BLZB0B](https://discuss.elastic.co/u/BLZB0B)\
**Post date:** [June 19, 2017, 4:11pm UTC](https://discuss.elastic.co/t/when-using-inner-hits-on-nested-query-we-are-getting-an-index-out-of-bounds-exception/89968/1 "2017-06-19T16:11:57Z")

</div>

Hi,

I'm pulling my hair out on this (and I don't have much left to pull out!)

We have a set of documents which are replicated to Elastic from Couchbase.

We're new to Elastic, so feel free to say "Why are you doing this, this is a better way" (so long as it's constructive 😉 )

in the document, we have a "holiday product", this product has a nested array of "prices", this price object contains fields such as number of passengers, date, promo code, price etc. For each holiday product, this list of prices can be very long as there can be hundreds/thousands of permutations.

When running a query against the data, we don't want to carry the 1000's of lines of data over the wire (can be \>5MB) so are trying to use inner\_hits to only return the rows that match. (approx 10KB)

we have a query such as:

```auto
{
"_source": false,
   "query": {
    "nested": {
      "path": "doc.products.calculatedPrices",
      "score_mode": "avg",
        "query": {
        "bool": {
          "must": [
            { "match": { "doc.products.calculatedPrices.promoCode": "DRH725C" } },
            { "match": { "doc.products.calculatedPrices.pax": 2 } },
            { "range": { "doc.products.calculatedPrices.now" : {"gte": 100, "lte":120} } }
           ]
        }    
	  },
	  "inner_hits": {}
    }
  }
}

```

this generates the error:

```auto
{
    "took": 41,
    "timed_out": false,
    "_shards": {
        "total": 3,
        "successful": 2,
        "failed": 1,
        "failures": [
            {
                "shard": 2,
                "index": "testavail5",
                "node": "AfXKKfqrSPqOrNO7mLreEQ",
                "reason": {
                    "type": "index_out_of_bounds_exception",
                    "reason": "Index: 7290, Size: 134"
                }
            }
        ]
    },
    "hits": {
        "total": 62,
        "max_score": 5.845086,
        "hits": []
    }
}

```

On this test server, we are running a single node with 3 shards. so not sure why one share reports as failed.

if we change the  
`inner_hits": {}` to `inner_hits": {"_source":false}`

we get results but obviously don't get any useful information in the output!

Are we hitting a bug in ES 5.4 ? (or more likely) are we doing something stupid?

Thanks

Phil

(happy to post more snippets / logs etc as required - just didn't want to overload the post with too much info that might not be needed)

---

<div class="post-metadata">

**Author:** ![mvg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvg/32/98890_2.png) [@mvg](https://discuss.elastic.co/u/mvg)\
**Post date:** [June 20, 2017, 8:26am UTC](https://discuss.elastic.co/t/when-using-inner-hits-on-nested-query-we-are-getting-an-index-out-of-bounds-exception/89968/2 "2017-06-20T08:26:24Z")

</div>

@BLZB0B I think this is a bug too. Do you see any stacktrace with the `Fetch phase failed` in elasticsearch's log file?

If you don't see any stacktraces, can you then add the following to the `log4j2.properties` file?

```auto
logger.search.name = org.elasticsearch.search
logger.search.level = trace

```

This should make the stacktrace visible. If you like you can also open an issue on github, since this is a bug.

---

<div class="post-metadata">

**Author:** ![BLZB0B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blzb0b/32/24714_2.png) [@BLZB0B](https://discuss.elastic.co/u/BLZB0B)\
**Post date:** [June 20, 2017, 11:28am UTC](https://discuss.elastic.co/t/when-using-inner-hits-on-nested-query-we-are-getting-an-index-out-of-bounds-exception/89968/3 "2017-06-20T11:28:21Z")

</div>

Hi Martijn,

we get the following:

```auto
[2017-06-20T11:46:58,972][TRACE][o.e.s.SearchService] [AfXKKfq] Fetch phase failed
java.lang.IndexOutOfBoundsException: Index: 7290, Size: 134
        at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[?:1.8.0_131]
        at java.util.ArrayList.get(ArrayList.java:429) ~[?:1.8.0_131]
        at org.elasticsearch.search.fetch.FetchPhase.createNestedSearchHit(FetchPhase.java:256) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:150) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.search.fetch.subphase.InnerHitsFetchSubPhase.hitExecute(InnerHitsFetchSubPhase.java:65) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:161) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:417) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.action.search.SearchTransportService$12.messageReceived(SearchTransportService.java:394) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.action.search.SearchTransportService$12.messageReceived(SearchTransportService.java:391) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:627) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.4.0.jar:5.4.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
[2017-06-20T11:46:58,973][DEBUG][o.e.a.s.TransportSearchAction] [AfXKKfq] [6] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [AfXKKfq][172.31.35.8:9300][indices:data/read/search[phase/fetch/id]]
Caused by: java.lang.IndexOutOfBoundsException: Index: 7290, Size: 134
        at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[?:1.8.0_131]
        at java.util.ArrayList.get(ArrayList.java:429) ~[?:1.8.0_131]
        at org.elasticsearch.search.fetch.FetchPhase.createNestedSearchHit(FetchPhase.java:256) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:150) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.search.fetch.subphase.InnerHitsFetchSubPhase.hitExecute(InnerHitsFetchSubPhase.java:65) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:161) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:417) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.action.search.SearchTransportService$12.messageReceived(SearchTransportService.java:394) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.action.search.SearchTransportService$12.messageReceived(SearchTransportService.java:391) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:627) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.4.0.jar:5.4.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

```

Thanks

Phil

---

<div class="post-metadata">

**Author:** ![mvg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvg/32/98890_2.png) [@mvg](https://discuss.elastic.co/u/mvg)\
**Post date:** [June 20, 2017, 1:22pm UTC](https://discuss.elastic.co/t/when-using-inner-hits-on-nested-query-we-are-getting-an-index-out-of-bounds-exception/89968/4 "2017-06-20T13:22:09Z")

</div>

Thanks Phil, this is very helpful.

---

<div class="post-metadata">

**Author:** ![BLZB0B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blzb0b/32/24714_2.png) [@BLZB0B](https://discuss.elastic.co/u/BLZB0B)\
**Post date:** [June 20, 2017, 2:24pm UTC](https://discuss.elastic.co/t/when-using-inner-hits-on-nested-query-we-are-getting-an-index-out-of-bounds-exception/89968/5 "2017-06-20T14:24:41Z")

</div>

Hi Martijn,

Github bug raised as suggested : [https://github.com/elastic/elasticsearch/issues/25315](https://github.com/elastic/elasticsearch/issues/25315)

are there any good work arounds that you could suggest? "all we need to do" is to reduce the inner results of the array before returning the data.

Thanks

Phil

---

<div class="post-metadata">

**Author:** ![mvg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvg/32/98890_2.png) [@mvg](https://discuss.elastic.co/u/mvg)\
**Post date:** [June 20, 2017, 2:38pm UTC](https://discuss.elastic.co/t/when-using-inner-hits-on-nested-query-we-are-getting-an-index-out-of-bounds-exception/89968/6 "2017-06-20T14:38:17Z")

</div>

What you can try to do as a workaround is to use doc\_values fields inside the inner hit definition:  
[https://www.elastic.co/guide/en/elasticsearch/reference/5.4/search-request-docvalue-fields.html](https://www.elastic.co/guide/en/elasticsearch/reference/5.4/search-request-docvalue-fields.html)

Most fields nowadays have doc values enabled by default (fields mapped as text don't have doc values), so you should be able to return most of your nested fields.

---

<div class="post-metadata">

**Author:** ![BLZB0B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blzb0b/32/24714_2.png) [@BLZB0B](https://discuss.elastic.co/u/BLZB0B)\
**Post date:** [June 20, 2017, 2:39pm UTC](https://discuss.elastic.co/t/when-using-inner-hits-on-nested-query-we-are-getting-an-index-out-of-bounds-exception/89968/7 "2017-06-20T14:39:01Z")

</div>

thanks - will take a look now.

Regards

Phil

---

<div class="post-metadata">

**Author:** ![BLZB0B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blzb0b/32/24714_2.png) [@BLZB0B](https://discuss.elastic.co/u/BLZB0B)\
**Post date:** [June 20, 2017, 3:11pm UTC](https://discuss.elastic.co/t/when-using-inner-hits-on-nested-query-we-are-getting-an-index-out-of-bounds-exception/89968/8 "2017-06-20T15:11:49Z")

</div>

Hi @mvg ,

Unfortunately, I don't think this is an option as we're running nested queries:

```auto
[nested] query does not support [docvalue_fields]

```

☹

Regards

Phil

---

<div class="post-metadata">

**Author:** ![mvg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvg/32/98890_2.png) [@mvg](https://discuss.elastic.co/u/mvg)\
**Post date:** [June 20, 2017, 3:14pm UTC](https://discuss.elastic.co/t/when-using-inner-hits-on-nested-query-we-are-getting-an-index-out-of-bounds-exception/89968/9 "2017-06-20T15:14:46Z")

</div>

Can you share the query you were trying to execute? Looks like `docvalue_fields` isn't specified in the `inner_hits` itself, but directly in the `nested` query.

---

<div class="post-metadata">

**Author:** ![BLZB0B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blzb0b/32/24714_2.png) [@BLZB0B](https://discuss.elastic.co/u/BLZB0B)\
**Post date:** [June 20, 2017, 3:24pm UTC](https://discuss.elastic.co/t/when-using-inner-hits-on-nested-query-we-are-getting-an-index-out-of-bounds-exception/89968/10 "2017-06-20T15:24:05Z")

</div>

@mvg Sorry, built a dodgy query...

corrected it but not quite getting the expected results:

```auto
{
"_source": false,
   "query": {
    "nested": {
      "path": "doc.products.calculatedPrices",
      "score_mode": "avg",
        "query": {
        "bool": {
          "must": [
            { "match": { "doc.products.calculatedPrices.promoCode": "DRH725C" } },
            { "match": { "doc.products.calculatedPrices.pax": 2 } },
            { "range": { "doc.products.calculatedPrices.now" : {"gte": 100, "lte":120} } }
           ]
        }
      
	  },
	  "inner_hits":{ // The actual list would be slightly longer but just these fields will be fine for the moment
                "stored_fields" : ["doc.products.calculatedPrices.was","doc.products.calculatedPrices.now"]
	  }
    }
  }
}

```

returns (snippet):

```auto
{
    "took": 106,
    "timed_out": false,
    "_shards": {
        "total": 3,
        "successful": 3,
        "failed": 0
    },
    "hits": {
        "total": 60,
        "max_score": 5.831132,
        "hits": [
            {
                "_index": "testavail5",
                "_type": "couchbaseDocument",
                "_id": "CLMOEA2",
                "_score": 5.831132,
                "inner_hits": {
                    "doc.products.calculatedPrices": {
                        "hits": {
                            "total": 42,
                            "max_score": 5.831132,
                            "hits": [
                                {
                                    "_nested": {
                                        "field": "doc.products.calculatedPrices",
                                        "offset": 11087
                                    },
                                    "_score": 5.831132
                                },
                                {
                                    "_nested": {
                                        "field": "doc.products.calculatedPrices",
                                        "offset": 10995
                                    },
                                    "_score": 5.831132
                                },
                                {
                                    "_nested": {
                                        "field": "doc.products.calculatedPrices",
                                        "offset": 10365
                                    },
                                    "_score": 5.831132
                                }
                            ]
                        }
                    }
                }
            }, <snip>

```

thanks

Phil

---

<div class="post-metadata">

**Author:** ![mvg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvg/32/98890_2.png) [@mvg](https://discuss.elastic.co/u/mvg)\
**Post date:** [June 20, 2017, 3:34pm UTC](https://discuss.elastic.co/t/when-using-inner-hits-on-nested-query-we-are-getting-an-index-out-of-bounds-exception/89968/11 "2017-06-20T15:34:53Z")

</div>

That is because `stored_fields` is used instead of `docvalue_fields`, can you try this:

```auto
{
"_source": false,
   "query": {
    "nested": {
      "path": "doc.products.calculatedPrices",
      "score_mode": "avg",
        "query": {
        "bool": {
          "must": [
            { "match": { "doc.products.calculatedPrices.promoCode": "DRH725C" } },
            { "match": { "doc.products.calculatedPrices.pax": 2 } },
            { "range": { "doc.products.calculatedPrices.now" : {"gte": 100, "lte":120} } }
           ]
        }
      
	  },
	  "inner_hits":{ // The actual list would be slightly longer but just these fields will be fine for the moment
                "docvalue_fields" : ["doc.products.calculatedPrices.was","doc.products.calculatedPrices.now"]
	  }
    }
  }

```

Stored fields are disabled by default (except for `_source` field), so it is expected that you don't see any key value pairs.

---

<div class="post-metadata">

**Author:** ![BLZB0B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blzb0b/32/24714_2.png) [@BLZB0B](https://discuss.elastic.co/u/BLZB0B)\
**Post date:** [June 20, 2017, 3:39pm UTC](https://discuss.elastic.co/t/when-using-inner-hits-on-nested-query-we-are-getting-an-index-out-of-bounds-exception/89968/12 "2017-06-20T15:39:43Z")

</div>

@mvg

hi,

That takes us back to the "index\_out\_of\_bounds\_exception" again ☹

```auto
[2017-06-20T15:37:40,783][TRACE][o.e.s.SearchService] [AfXKKfq] Fetch phase failed
java.lang.IndexOutOfBoundsException: Index: 11087, Size: 77
        at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[?:1.8.0_131]
        at java.util.ArrayList.get(ArrayList.java:429) ~[?:1.8.0_131]
        at org.elasticsearch.search.fetch.FetchPhase.createNestedSearchHit(FetchPhase.java:256) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:150) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.search.fetch.subphase.InnerHitsFetchSubPhase.hitExecute(InnerHitsFetchSubPhase.java:65) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:161) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:417) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.action.search.SearchTransportService$12.messageReceived(SearchTransportService.java:394) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.action.search.SearchTransportService$12.messageReceived(SearchTransportService.java:391) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:627) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.4.0.jar:5.4.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
[2017-06-20T15:37:40,784][DEBUG][o.e.a.s.TransportSearchAction] [AfXKKfq] [1014] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [AfXKKfq][172.31.35.8:9300][indices:data/read/search[phase/fetch/id]]
Caused by: java.lang.IndexOutOfBoundsException: Index: 11087, Size: 77
        at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[?:1.8.0_131]
        at java.util.ArrayList.get(ArrayList.java:429) ~[?:1.8.0_131]
        at org.elasticsearch.search.fetch.FetchPhase.createNestedSearchHit(FetchPhase.java:256) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:150) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.search.fetch.subphase.InnerHitsFetchSubPhase.hitExecute(InnerHitsFetchSubPhase.java:65) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:161) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:417) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.action.search.SearchTransportService$12.messageReceived(SearchTransportService.java:394) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.action.search.SearchTransportService$12.messageReceived(SearchTransportService.java:391) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:627) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.4.0.jar:5.4.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

```

Appreciate your patience and help on this...

---

<div class="post-metadata">

**Author:** ![mvg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvg/32/98890_2.png) [@mvg](https://discuss.elastic.co/u/mvg)\
**Post date:** [June 20, 2017, 3:42pm UTC](https://discuss.elastic.co/t/when-using-inner-hits-on-nested-query-we-are-getting-an-index-out-of-bounds-exception/89968/13 "2017-06-20T15:42:39Z")

</div>

Oops, the nested source fetching needs to be disabled (by adding `"_source": false` to `inner_hits`):

```auto
{
"_source": false,
   "query": {
    "nested": {
      "path": "doc.products.calculatedPrices",
      "score_mode": "avg",
        "query": {
        "bool": {
          "must": [
            { "match": { "doc.products.calculatedPrices.promoCode": "DRH725C" } },
            { "match": { "doc.products.calculatedPrices.pax": 2 } },
            { "range": { "doc.products.calculatedPrices.now" : {"gte": 100, "lte":120} } }
           ]
        }
      
	  },
	  "inner_hits":{ // The actual list would be slightly longer but just these fields will be fine for the moment
                "docvalue_fields" : ["doc.products.calculatedPrices.was","doc.products.calculatedPrices.now"],
                "_source": false
	  }
    }
  }

```

---

<div class="post-metadata">

**Author:** ![BLZB0B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blzb0b/32/24714_2.png) [@BLZB0B](https://discuss.elastic.co/u/BLZB0B)\
**Post date:** [June 20, 2017, 3:52pm UTC](https://discuss.elastic.co/t/when-using-inner-hits-on-nested-query-we-are-getting-an-index-out-of-bounds-exception/89968/14 "2017-06-20T15:52:20Z")

</div>

@mvg

haha... Yes 😄

getting results now!

I'll expand the above example to include additional fields so the results make more sense - I'll need to update our mapping to include keywords on some of the text fields

```auto
"Fielddata is disabled on text fields by default. Set fielddata=true on [doc.products.calculatedPrices.promoCode] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."

```

but that should be simple enough (he says)

Thanks again

Phil

---

<div class="post-metadata">

**Author:** ![BLZB0B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blzb0b/32/24714_2.png) [@BLZB0B](https://discuss.elastic.co/u/BLZB0B)\
**Post date:** [June 22, 2017, 4:30pm UTC](https://discuss.elastic.co/t/when-using-inner-hits-on-nested-query-we-are-getting-an-index-out-of-bounds-exception/89968/15 "2017-06-22T16:30:17Z")

</div>

Hi Martijn,

thanks for your help so far - the last issue (for the moment) is as follows.

You've probably seen in our data, we have a doc, which has nested products, which have nested calculatedPrices.

when we run a query, the inner hits for calculatedPrices need to be at the product level not at the doc level eg:

```auto
  "hits": {
        "total": 241,
        "max_score": 4.94516,
        "hits": [
            {
                "_index": "testavail5",
                "_type": "couchbaseDocument",
                "_id": "COHARA2",
                "_score": 4.94516,
                "_source": {
                    "doc": {
                        "destinationCode": "HAR",
                        "productSku": "Harrogate, St George Hotel",
                        "productType": "SelfDrive"
                    }
                },
                "inner_hits": {
                    "doc.products.calculatedPrices": {
                        "hits": {
                            "total": 198,
                            "max_score": 4.94516,
                            "hits": [
                                {
                                    "_nested": {
                                        "field": "doc.products",
                                        "offset": 32,
                                        "_nested": {
                                            "field": "calculatedPrices",
                                            "offset": 101
                                        }
                                    },
                                    <blah blah>

```

is incorrect... as the inner hits are specific to a particular product within the doc.

it needs to be something _like_ but not exaclty :

```auto
  "hits": {
        "total": 241,
        "max_score": 4.94516,
        "hits": [
            {
                "_index": "testavail5",
                "_type": "couchbaseDocument",
                "_id": "COHARA2",
                "_score": 4.94516,
                "_source": {
                    "doc": {
                        "destinationCode": "HAR",
                        "productSku": "Harrogate, St George Hotel",
                        "productType": "SelfDrive",
						"inner_hits": {
						"doc.products.calculatedPrices": {
							"hits": {
								"total": 198,
								"max_score": 4.94516,
								"hits": [
									{
										"_nested": {
											"field": "doc.products",
											"offset": 32,
											"_nested": {
												"field": "calculatedPrices",
												"offset": 101
											}
										},
										<blah blah>

```

is this even possible? any advice / pointers / documentation to read would be much appreciated.

Regards

Phil

---

<div class="post-metadata">

**Author:** ![mvg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvg/32/98890_2.png) [@mvg](https://discuss.elastic.co/u/mvg)\
**Post date:** [June 23, 2017, 8:06am UTC](https://discuss.elastic.co/t/when-using-inner-hits-on-nested-query-we-are-getting-an-index-out-of-bounds-exception/89968/16 "2017-06-23T08:06:42Z")

</div>

Hi Phil,

The reason the inner hit is directly returned `doc.products.calculatedPrices` level is because `doc` and `products` fields are object fields.

I think the the structure of the hits will be better if you make `products` field nested too. By doing that you will need to use the `nested` query more in order to query the product properties.

Your query will then look like this:

```auto
{
  "_source": false,
  "query": {
    "nested": {
      "path": "doc.products",
      "score_mode": "avg",
      "inner_hits": {
        "_source": false
      },
      "query": {
        "nested": {
          "path": "doc.products.calculatedPrices",
          "score_mode": "avg",
          "query": {
            "bool": {
              "must": [
                {
                  "match": {
                    "doc.products.calculatedPrices.promoCode": "DRH725C"
                  }
                },
                {
                  "match": {
                    "doc.products.calculatedPrices.pax": 2
                  }
                },
                {
                  "range": {
                    "doc.products.calculatedPrices.now": {
                      "gte": 100,
                      "lte": 120
                    }
                  }
                }
              ]
            }
          },
          "inner_hits": {
            "docvalue_fields": [
              "doc.products.calculatedPrices.was",
              "doc.products.calculatedPrices.now"
            ],
            "_source": false
          }
        }
      }
    }
  }
}

```

I hope this helps.

Martijn

---

<div class="post-metadata">

**Author:** ![BLZB0B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blzb0b/32/24714_2.png) [@BLZB0B](https://discuss.elastic.co/u/BLZB0B)\
**Post date:** [June 23, 2017, 9:29am UTC](https://discuss.elastic.co/t/when-using-inner-hits-on-nested-query-we-are-getting-an-index-out-of-bounds-exception/89968/17 "2017-06-23T09:29:59Z")

</div>

Many thanks,

I will give this a try shortly. We have already set products as nested in the mapping based on your previous suggestions so we should be in a good place to try the above.

Many thanks again..

Phil

---

<div class="post-metadata">

**Author:** ![BLZB0B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blzb0b/32/24714_2.png) [@BLZB0B](https://discuss.elastic.co/u/BLZB0B)\
**Post date:** [July 20, 2017, 11:25am UTC](https://discuss.elastic.co/t/when-using-inner-hits-on-nested-query-we-are-getting-an-index-out-of-bounds-exception/89968/18 "2017-07-20T11:25:13Z")

</div>

Hi Martijn,

We're still building this ☹ what we've noticed is that as the amount of nested data increases, there is a huge slowdown in the results. I'm guessing this is to be expected to a certain extent - but it seems to be an exponential slow down.

Based on what you know of our data - would reshaping the data be beneficial? would reducing from 2 levels of nesting to one be beneficial? would flattening all of the data from 6000 "rows" to 20m "rows" have it's own performance challenges ?

thanks

Phil

---

<div class="post-metadata">

**Author:** ![mvg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvg/32/98890_2.png) [@mvg](https://discuss.elastic.co/u/mvg)\
**Post date:** [July 21, 2017, 6:59am UTC](https://discuss.elastic.co/t/when-using-inner-hits-on-nested-query-we-are-getting-an-index-out-of-bounds-exception/89968/19 "2017-07-21T06:59:41Z")

</div>

Hi Phil,

I think flatting the data will for sure have a positive impact on the performance of your queries. For example I think that the doc nested field doesn't add a whole lot of meaning, so I would remove that nested level. You can also duplicate the product data in each price nested object, but that is going to increase the amount of document like you said. ES should be able to handle this fine, but it does mean that your application needs to handle the search responses of certain queries differently (data isn't structured the same any more).

Maybe before going down this path, it would be good to know what is exactly slow. If you run your queries in a continuous loop and at the same time capture [hot threads](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/cluster-nodes-hot-threads.html) then I'll be happy to look at it.

Martijn

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 18, 2017, 6:59am UTC](https://discuss.elastic.co/t/when-using-inner-hits-on-nested-query-we-are-getting-an-index-out-of-bounds-exception/89968/20 "2017-08-18T06:59:44Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.