Error retrieving path of nested field

Hello, we are experiencing issues with one query in particular, however this is only happening ocassionally and we can't figure out the source of the problem.

The query is working fine

get homes/_search
{
  "size": 0,
  "query": {
    "bool": {
      "filter": [
        {
          "match": {
            "location_2": 5168
          }
        },
        {
          "match": {
            "operation": "sale"
          }
        },
        {
          "nested": {
            "path": "offers",
            "inner_hits": {
              "_source": [
                "offers.id"
              ]
            },
            "query": {
              "bool": {
                "filter": [
                  {
                    "match": {
                      "offers.location_2": 5168
                    }
                  },
                  {
                    "match": {
                      "offers.operation": "sale"
                    }
                  }
                ]
              }
            }
          }
        },
        {
          "geo_shape": {
            "location": {
              "shape": """
          POLYGON((
            -5.847290754318238 43.359096194564316,
            -5.842151641845704 43.359096194564316,
            -5.842151641845704 43.355542934897514,
            -5.847290754318238 43.355542934897514,
            -5.847290754318238 43.359096194564316
          ))      
          """,
              "relation": "within"
            }
          }
        }
      ]
    }
  },
  "aggs": {
    "clusters": {
      "geotile_grid": {
        "field": "location",
        "precision": 20
      },
      "aggs": {
        "hits": {
          "top_hits": {
            "size": 100,
            "_source": [
              "offers.id"            ]
          }
        }
      }
    }
  }
}

However this one:

get homes/_search
{
  "size": 0,
  "query": {
    "bool": {
      "filter": [
        {
          "match": {
            "location_2": 5168
          }
        },
        {
          "match": {
            "operation": "sale"
          }
        },
        {
          "nested": {
            "path": "offers",
            "inner_hits": {
              "_source": [
                "offers.id"
              ]
            },
            "query": {
              "bool": {
                "filter": [
                  {
                    "match": {
                      "offers.location_2": 5168
                    }
                  },
                  {
                    "match": {
                      "offers.operation": "sale"
                    }
                  }
                  
                ]
              }
            }
          }
        },
        {
          "geo_shape": {
            "location": {
              "shape": """
          POLYGON((
            -5.8480310440063485 43.36549231759247,
                    -5.83775281906128 43.36549231759247,
                    -5.83775281906128 43.35838633934203,
                    -5.8480310440063485 43.35838633934203,
                    -5.8480310440063485 43.36549231759247
          ))      
          """,
              "relation": "within"
            }
          }
        }
      ]
    }
  },
  "aggs": {
    "clusters": {
      "geotile_grid": {
        "field": "location",
        "precision": 20
      },
      "aggs": {
        "hits": {
          "top_hits": {
            "size": 100,
            "_source": [
              "offers.id"
            ]
          }
        }
      }
    }
  }
}

Gives the following error

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_state_exception",
        "reason": "Error retrieving path offers"
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": "homes",
        "node": "4lIopJZcRAGJHd_DnSlSeg",
        "reason": {
          "type": "illegal_state_exception",
          "reason": "Error retrieving path offers"
        }
      }
    ],
    "caused_by": {
      "type": "illegal_state_exception",
      "reason": "Error retrieving path offers",
      "caused_by": {
        "type": "illegal_state_exception",
        "reason": "Error retrieving path offers"
      }
    }
  },
  "status": 500
}

The field "offers" is mapped as nested. We also have other index with similar mappings where the queries work without any errors or warnings.

Any idea what the issue can be?

If you re-run the query with ?error_trace what is the output?

Hello! Thanks for answering, this is the output:

  "error": {
    "root_cause": [
      {
        "type": "illegal_state_exception",
        "reason": "Error retrieving path offers",
        "stack_trace": """org.elasticsearch.ElasticsearchException$1: Error retrieving path offers
	at org.elasticsearch.server@8.13.4/org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:701)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.search.SearchPhaseExecutionException.guessRootCauses(SearchPhaseExecutionException.java:160)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:683)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.ElasticsearchException.generateFailureXContent(ElasticsearchException.java:629)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.rest.RestResponse.build(RestResponse.java:184)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.rest.RestResponse.<init>(RestResponse.java:143)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.rest.RestResponse.<init>(RestResponse.java:114)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.rest.action.RestActionListener.onFailure(RestActionListener.java:55)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.rest.action.RestCancellableNodeClient$1.onFailure(RestCancellableNodeClient.java:96)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.tasks.TaskManager$1.onFailure(TaskManager.java:214)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:31)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onFailure(ActionListenerImplementations.java:317)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:31)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:39)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:31)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:31)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:31)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.ActionListenerImplementations$RunAfterActionListener.onFailure(ActionListenerImplementations.java:278)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.search.AbstractSearchAsyncAction.raisePhaseFailure(AbstractSearchAsyncAction.java:735)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:712)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:404)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:744)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:497)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:335)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:31)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:53)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:634)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.handleException(TransportService.java:1751)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1475)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1609)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1584)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:44)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:44)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:146)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.IllegalStateException: Error retrieving path offers
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.SearchHit$NestedIdentity.extractSource(SearchHit.java:1324)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.prepareNestedHitContext(FetchPhase.java:336)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.prepareHitContext(FetchPhase.java:222)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:155)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:70)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:175)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:78)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.subphase.InnerHitsPhase.hitExecute(InnerHitsPhase.java:95)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.subphase.InnerHitsPhase$1.process(InnerHitsPhase.java:62)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:169)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:70)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:175)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:78)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.metrics.TopHitsAggregator.runFetchPhase(TopHitsAggregator.java:228)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.metrics.TopHitsAggregator.buildAggregation(TopHitsAggregator.java:195)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.metrics.MetricsAggregator.buildAggregations(MetricsAggregator.java:41)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:180)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForAllBuckets(BucketsAggregator.java:224)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.bucket.geogrid.GeoGridAggregator.buildAggregations(GeoGridAggregator.java:160)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
	at org.apache.lucene.core@9.10.0/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:117)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 6 more
"""
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": "homes",
        "node": "4lIopJZcRAGJHd_DnSlSeg",
        "reason": {
          "type": "illegal_state_exception",
          "reason": "Error retrieving path offers",
          "stack_trace": """java.lang.IllegalStateException: Error retrieving path offers
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.SearchHit$NestedIdentity.extractSource(SearchHit.java:1324)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.prepareNestedHitContext(FetchPhase.java:336)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.prepareHitContext(FetchPhase.java:222)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:155)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:70)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:175)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:78)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.subphase.InnerHitsPhase.hitExecute(InnerHitsPhase.java:95)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.subphase.InnerHitsPhase$1.process(InnerHitsPhase.java:62)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:169)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:70)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:175)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:78)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.metrics.TopHitsAggregator.runFetchPhase(TopHitsAggregator.java:228)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.metrics.TopHitsAggregator.buildAggregation(TopHitsAggregator.java:195)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.metrics.MetricsAggregator.buildAggregations(MetricsAggregator.java:41)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:180)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForAllBuckets(BucketsAggregator.java:224)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.bucket.geogrid.GeoGridAggregator.buildAggregations(GeoGridAggregator.java:160)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
	at org.apache.lucene.core@9.10.0/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:117)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
"""
        }
      }
    ],
    "caused_by": {
      "type": "illegal_state_exception",
      "reason": "Error retrieving path offers",
      "caused_by": {
        "type": "illegal_state_exception",
        "reason": "Error retrieving path offers",
        "stack_trace": """java.lang.IllegalStateException: Error retrieving path offers
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.SearchHit$NestedIdentity.extractSource(SearchHit.java:1324)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.prepareNestedHitContext(FetchPhase.java:336)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.prepareHitContext(FetchPhase.java:222)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:155)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:70)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:175)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:78)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.subphase.InnerHitsPhase.hitExecute(InnerHitsPhase.java:95)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.subphase.InnerHitsPhase$1.process(InnerHitsPhase.java:62)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:169)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:70)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:175)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:78)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.metrics.TopHitsAggregator.runFetchPhase(TopHitsAggregator.java:228)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.metrics.TopHitsAggregator.buildAggregation(TopHitsAggregator.java:195)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.metrics.MetricsAggregator.buildAggregations(MetricsAggregator.java:41)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:180)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForAllBuckets(BucketsAggregator.java:224)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.bucket.geogrid.GeoGridAggregator.buildAggregations(GeoGridAggregator.java:160)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
	at org.apache.lucene.core@9.10.0/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:117)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
"""
      },
      "stack_trace": """org.elasticsearch.ElasticsearchException$1: Error retrieving path offers
	at org.elasticsearch.server@8.13.4/org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:701)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:402)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:744)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:497)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:335)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:31)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:53)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:634)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.handleException(TransportService.java:1751)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1475)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1609)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1584)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:44)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:44)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:146)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.IllegalStateException: Error retrieving path offers
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.SearchHit$NestedIdentity.extractSource(SearchHit.java:1324)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.prepareNestedHitContext(FetchPhase.java:336)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.prepareHitContext(FetchPhase.java:222)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:155)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:70)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:175)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:78)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.subphase.InnerHitsPhase.hitExecute(InnerHitsPhase.java:95)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.subphase.InnerHitsPhase$1.process(InnerHitsPhase.java:62)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:169)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:70)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:175)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:78)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.metrics.TopHitsAggregator.runFetchPhase(TopHitsAggregator.java:228)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.metrics.TopHitsAggregator.buildAggregation(TopHitsAggregator.java:195)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.metrics.MetricsAggregator.buildAggregations(MetricsAggregator.java:41)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:180)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForAllBuckets(BucketsAggregator.java:224)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.bucket.geogrid.GeoGridAggregator.buildAggregations(GeoGridAggregator.java:160)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
	at org.apache.lucene.core@9.10.0/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:117)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 6 more
"""
    },

Last part of the trace as it was too long to post

"""
    },
    "stack_trace": """Failed to execute phase [query], all shards failed; shardFailures {[NBz9DWz_QxuEOZHWnGZtCA][homes][0]: org.elasticsearch.transport.RemoteTransportException: [instance-0000000030][10.43.0.63:19214][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.query.QueryPhaseExecutionException: Query Failed [Failed to execute main query]
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.query.QueryPhase.addCollectorsAndSearch(QueryPhase.java:229)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.query.QueryPhase.executeQuery(QueryPhase.java:135)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:63)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$31(IndicesService.java:1529)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$32(IndicesService.java:1595)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:164)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:147)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:418)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:110)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1601)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1523)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:520)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:680)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:549)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:73)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:70)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:95)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: org.elasticsearch.search.fetch.FetchPhaseExecutionException: Fetch Failed [Error running fetch phase for doc [28828556]]
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:78)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:175)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:78)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.metrics.TopHitsAggregator.runFetchPhase(TopHitsAggregator.java:228)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.metrics.TopHitsAggregator.buildAggregation(TopHitsAggregator.java:195)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.metrics.MetricsAggregator.buildAggregations(MetricsAggregator.java:41)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:180)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForAllBuckets(BucketsAggregator.java:224)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.bucket.geogrid.GeoGridAggregator.buildAggregations(GeoGridAggregator.java:160)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
	at org.apache.lucene.core@9.10.0/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:117)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 6 more
	Suppressed: org.elasticsearch.search.fetch.FetchPhaseExecutionException: Fetch Failed [Error running fetch phase for doc [16818803]]
		... 23 more
	Caused by: org.elasticsearch.search.fetch.FetchPhaseExecutionException: Fetch Failed [Error running fetch phase for doc [16818801]]
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:78)
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:175)
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:78)
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.subphase.InnerHitsPhase.hitExecute(InnerHitsPhase.java:95)
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.subphase.InnerHitsPhase$1.process(InnerHitsPhase.java:62)
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:169)
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:70)
		... 22 more
	Caused by: java.lang.IllegalStateException: Error retrieving path offers
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.SearchHit$NestedIdentity.extractSource(SearchHit.java:1324)
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.prepareNestedHitContext(FetchPhase.java:336)
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.prepareHitContext(FetchPhase.java:222)
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:155)
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:70)
		... 28 more
	Suppressed: org.elasticsearch.search.fetch.FetchPhaseExecutionException: Fetch Failed [Error running fetch phase for doc [3114279]]
		... 23 more
	Caused by: org.elasticsearch.search.fetch.FetchPhaseExecutionException: Fetch Failed [Error running fetch phase for doc [3114277]]
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:78)
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:175)
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:78)
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.subphase.InnerHitsPhase.hitExecute(InnerHitsPhase.java:95)
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.subphase.InnerHitsPhase$1.process(InnerHitsPhase.java:62)
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:169)
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:70)
		... 22 more
	Caused by: java.lang.IllegalStateException: Error retrieving path offers
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.SearchHit$NestedIdentity.extractSource(SearchHit.java:1324)
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.prepareNestedHitContext(FetchPhase.java:336)
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.prepareHitContext(FetchPhase.java:222)
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:155)
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:70)
		... 28 more
Caused by: org.elasticsearch.search.fetch.FetchPhaseExecutionException: Fetch Failed [Error running fetch phase for doc [28828552]]
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:78)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:175)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:78)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.subphase.InnerHitsPhase.hitExecute(InnerHitsPhase.java:95)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.subphase.InnerHitsPhase$1.process(InnerHitsPhase.java:62)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:169)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:70)
	... 22 more
Caused by: java.lang.IllegalStateException: Error retrieving path offers
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.SearchHit$NestedIdentity.extractSource(SearchHit.java:1324)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.prepareNestedHitContext(FetchPhase.java:336)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.prepareHitContext(FetchPhase.java:222)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:155)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:70)
	... 28 more
}{[NBz9DWz_QxuEOZHWnGZtCA][homes][1]: org.elasticsearch.transport.RemoteTransportException: [instance-0000000030][10.43.0.63:19214][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.query.QueryPhaseExecutionException: Query Failed [Failed to execute main query]
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.query.QueryPhase.addCollectorsAndSearch(QueryPhase.java:229)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.query.QueryPhase.executeQuery(QueryPhase.java:135)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:63)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$31(IndicesService.java:1529)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$32(IndicesService.java:1595)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:164)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:147)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:418)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:110)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1601)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1523)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:520)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:680)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:549)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:73)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:70)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:95)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: org.elasticsearch.search.fetch.FetchPhaseExecutionException: Fetch Failed [Error running fetch phase for doc [25323423]]
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:78)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:175)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:78)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.metrics.TopHitsAggregator.runFetchPhase(TopHitsAggregator.java:228)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.metrics.TopHitsAggregator.buildAggregation(TopHitsAggregator.java:195)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.metrics.MetricsAggregator.buildAggregations(MetricsAggregator.java:41)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:180)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForAllBuckets(BucketsAggregator.java:224)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.bucket.geogrid.GeoGridAggregator.buildAggregations(GeoGridAggregator.java:160)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
	at org.apache.lucene.core@9.10.0/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:117)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 6 more
	Suppressed: org.elasticsearch.search.fetch.FetchPhaseExecutionException: Fetch Failed [Error running fetch phase for doc [14558514]]
		... 23 more
	Caused by: org.elasticsearch.search.fetch.FetchPhaseExecutionException: Fetch Failed [Error running fetch phase for doc [14558505]]
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:78)
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:175)
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:78)
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.subphase.InnerHitsPhase.hitExecute(InnerHitsPhase.java:95)
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.subphase.InnerHitsPhase$1.process(InnerHitsPhase.java:62)
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:169)
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:70)
		... 22 more
	Caused by: java.lang.IllegalStateException: Error retrieving path offers
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.SearchHit$NestedIdentity.extractSource(SearchHit.java:1324)
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.prepareNestedHitContext(FetchPhase.java:336)
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.prepareHitContext(FetchPhase.java:222)
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:155)
		at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:70)
		... 28 more
Caused by: org.elasticsearch.search.fetch.FetchPhaseExecutionException: Fetch Failed [Error running fetch phase for doc [25323422]]
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:78)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:175)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:78)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.subphase.InnerHitsPhase.hitExecute(InnerHitsPhase.java:95)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.subphase.InnerHitsPhase$1.process(InnerHitsPhase.java:62)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:169)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:70)
	... 22 more
Caused by: java.lang.IllegalStateException: Error retrieving path offers
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.SearchHit$NestedIdentity.extractSource(SearchHit.java:1324)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.prepareNestedHitContext(FetchPhase.java:336)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.prepareHitContext(FetchPhase.java:222)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:155)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:70)
	... 28 more
}{[4lIopJZcRAGJHd_DnSlSeg][homes][2]: org.elasticsearch.transport.RemoteTransportException: [instance-0000000029][172.17.0.7:19599][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.query.QueryPhaseExecutionException: Query Failed [Failed to execute main query]
	at org.elasticsearch.search.query.QueryPhase.addCollectorsAndSearch(QueryPhase.java:229)
	at org.elasticsearch.search.query.QueryPhase.executeQuery(QueryPhase.java:135)
	at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:63)
	at org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$31(IndicesService.java:1529)
	at org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$32(IndicesService.java:1595)
	at org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:164)
	at org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:147)
	at org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:418)
	at org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:110)
	at org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1601)
	at org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1523)
	at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:520)
	at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:680)
	at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:549)
	at org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:73)
	at org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:70)
	at org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:95)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.lang.Thread.run(Thread.java:1583)
Caused by: org.elasticsearch.search.fetch.FetchPhaseExecutionException: Fetch Failed [Error running fetch phase for doc [11032339]]
	at org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:78)
	at org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:175)
	at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:78)
	at org.elasticsearch.search.aggregations.metrics.TopHitsAggregator.runFetchPhase(TopHitsAggregator.java:228)
	at org.elasticsearch.search.aggregations.metrics.TopHitsAggregator.buildAggregation(TopHitsAggregator.java:195)
	at org.elasticsearch.search.aggregations.metrics.MetricsAggregator.buildAggregations(MetricsAggregator.java:41)
	at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:180)
	at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForAllBuckets(BucketsAggregator.java:224)
	at org.elasticsearch.search.aggregations.bucket.geogrid.GeoGridAggregator.buildAggregations(GeoGridAggregator.java:160)
	at org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
	at org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
	at org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
	at org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
	at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
	at org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
	at org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:117)
	at java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 6 more
	Suppressed: org.elasticsearch.search.fetch.FetchPhaseExecutionException: Fetch Failed [Error running fetch phase for doc [26846149]]
		... 23 more
	Caused by: org.elasticsearch.search.fetch.FetchPhaseExecutionException: Fetch Failed [Error running fetch phase for doc [26846148]]
		at org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:78)
		at org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:175)
		at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:78)
		at org.elasticsearch.search.fetch.subphase.InnerHitsPhase.hitExecute(InnerHitsPhase.java:95)
		at org.elasticsearch.search.fetch.subphase.InnerHitsPhase$1.process(InnerHitsPhase.java:62)
		at org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:169)
		at org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:70)
		... 22 more
	Caused by: java.lang.IllegalStateException: Error retrieving path offers
		at org.elasticsearch.search.SearchHit$NestedIdentity.extractSource(SearchHit.java:1324)
		at org.elasticsearch.search.fetch.FetchPhase.prepareNestedHitContext(FetchPhase.java:336)
		at org.elasticsearch.search.fetch.FetchPhase.prepareHitContext(FetchPhase.java:222)
		at org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:155)
		at org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:70)
		... 28 more
	Suppressed: org.elasticsearch.search.fetch.FetchPhaseExecutionException: Fetch Failed [Error running fetch phase for doc [7563116]]
		... 23 more
	Caused by: org.elasticsearch.search.fetch.FetchPhaseExecutionException: Fetch Failed [Error running fetch phase for doc [7563110]]
		at org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:78)
		at org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:175)
		at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:78)
		at org.elasticsearch.search.fetch.subphase.InnerHitsPhase.hitExecute(InnerHitsPhase.java:95)
		at org.elasticsearch.search.fetch.subphase.InnerHitsPhase$1.process(InnerHitsPhase.java:62)
		at org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:169)
		at org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:70)
		... 22 more
	Caused by: java.lang.IllegalStateException: Error retrieving path offers
		at org.elasticsearch.search.SearchHit$NestedIdentity.extractSource(SearchHit.java:1324)
		at org.elasticsearch.search.fetch.FetchPhase.prepareNestedHitContext(FetchPhase.java:336)
		at org.elasticsearch.search.fetch.FetchPhase.prepareHitContext(FetchPhase.java:222)
		at org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:155)
		at org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:70)
		... 28 more
Caused by: org.elasticsearch.search.fetch.FetchPhaseExecutionException: Fetch Failed [Error running fetch phase for doc [11032338]]
	at org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:78)
	at org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:175)
	at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:78)
	at org.elasticsearch.search.fetch.subphase.InnerHitsPhase.hitExecute(InnerHitsPhase.java:95)
	at org.elasticsearch.search.fetch.subphase.InnerHitsPhase$1.process(InnerHitsPhase.java:62)
	at org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:169)
	at org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:70)
	... 22 more
Caused by: java.lang.IllegalStateException: Error retrieving path offers
	at org.elasticsearch.search.SearchHit$NestedIdentity.extractSource(SearchHit.java:1324)
	at org.elasticsearch.search.fetch.FetchPhase.prepareNestedHitContext(FetchPhase.java:336)
	at org.elasticsearch.search.fetch.FetchPhase.prepareHitContext(FetchPhase.java:222)
	at org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:155)
	at org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:70)
	... 28 more
}{[4lIopJZcRAGJHd_DnSlSeg][homes][3]: org.elasticsearch.transport.RemoteTransportException: [instance-0000000029][172.17.0.7:19599][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.query.QueryPhaseExecutionException: Query Failed [Failed to execute main query]
	at org.elasticsearch.search.query.QueryPhase.addCollectorsAndSearch(QueryPhase.java:229)
	at org.elasticsearch.search.query.QueryPhase.executeQuery(QueryPhase.java:135)
	at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:63)
	at org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$31(IndicesService.java:1529)
	at org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$32(IndicesService.java:1595)
	at org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:164)
	at org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:147)
	at org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:418)
	at org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:110)
	at org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1601)
	at org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1523)
	at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:520)
	at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:680)
	at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:549)
	at org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:73)
	at org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:70)
	at org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:95)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.lang.Thread.run(Thread.java:1583)
Caused by: org.elasticsearch.search.fetch.FetchPhaseExecutionException: Fetch Failed [Error running fetch phase for doc [2309745]]
	at org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:78)
	at org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:175)
	at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:78)
	at org.elasticsearch.search.aggregations.metrics.TopHitsAggregator.runFetchPhase(TopHitsAggregator.java:228)
	at org.elasticsearch.search.aggregations.metrics.TopHitsAggregator.buildAggregation(TopHitsAggregator.java:195)
	at org.elasticsearch.search.aggregations.metrics.MetricsAggregator.buildAggregations(MetricsAggregator.java:41)
	at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:180)
	at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForAllBuckets(BucketsAggregator.java:224)
	at org.elasticsearch.search.aggregations.bucket.geogrid.GeoGridAggregator.buildAggregations(GeoGridAggregator.java:160)
	at org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
	at org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
	at org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
	at org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
	at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
	at org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
	at org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:117)
	at java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 6 more
	Suppressed: org.elasticsearch.search.fetch.FetchPhaseExecutionException: Fetch Failed [Error running fetch phase for doc [16316646]]
		... 23 more
	Caused by: org.elasticsearch.search.fetch.FetchPhaseExecutionException: Fetch Failed [Error running fetch phase for doc [16316645]]
		at org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:78)
		at org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:175)
		at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:78)
		at org.elasticsearch.search.fetch.subphase.InnerHitsPhase.hitExecute(InnerHitsPhase.java:95)
		at org.elasticsearch.search.fetch.subphase.InnerHitsPhase$1.process(InnerHitsPhase.java:62)
		at org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:169)
		at org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:70)
		... 22 more
	Caused by: java.lang.IllegalStateException: Error retrieving path offers
		at org.elasticsearch.search.SearchHit$NestedIdentity.extractSource(SearchHit.java:1324)
		at org.elasticsearch.search.fetch.FetchPhase.prepareNestedHitContext(FetchPhase.java:336)
		at org.elasticsearch.search.fetch.FetchPhase.prepareHitContext(FetchPhase.java:222)
		at org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:155)
		at org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:70)
		... 28 more
Caused by: org.elasticsearch.search.fetch.FetchPhaseExecutionException: Fetch Failed [Error running fetch phase for doc [2309726]]
	at org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:78)
	at org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:175)
	at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:78)
	at org.elasticsearch.search.fetch.subphase.InnerHitsPhase.hitExecute(InnerHitsPhase.java:95)
	at org.elasticsearch.search.fetch.subphase.InnerHitsPhase$1.process(InnerHitsPhase.java:62)
	at org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:169)
	at org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:70)
	... 22 more
Caused by: java.lang.IllegalStateException: Error retrieving path offers
	at org.elasticsearch.search.SearchHit$NestedIdentity.extractSource(SearchHit.java:1324)
	at org.elasticsearch.search.fetch.FetchPhase.prepareNestedHitContext(FetchPhase.java:336)
	at org.elasticsearch.search.fetch.FetchPhase.prepareHitContext(FetchPhase.java:222)
	at org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:155)
	at org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:70)
	... 28 more
}
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:712)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:404)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:744)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:497)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:335)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:31)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:53)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:634)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.handleException(TransportService.java:1751)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1475)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1609)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1584)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:44)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:44)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:146)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: org.elasticsearch.ElasticsearchException$1: Error retrieving path offers
	at org.elasticsearch.server@8.13.4/org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:701)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:402)
	... 22 more
Caused by: java.lang.IllegalStateException: Error retrieving path offers
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.SearchHit$NestedIdentity.extractSource(SearchHit.java:1324)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.prepareNestedHitContext(FetchPhase.java:336)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.prepareHitContext(FetchPhase.java:222)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:155)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:70)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:175)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:78)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.subphase.InnerHitsPhase.hitExecute(InnerHitsPhase.java:95)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.subphase.InnerHitsPhase$1.process(InnerHitsPhase.java:62)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:169)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:70)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:175)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:78)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.metrics.TopHitsAggregator.runFetchPhase(TopHitsAggregator.java:228)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.metrics.TopHitsAggregator.buildAggregation(TopHitsAggregator.java:195)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.metrics.MetricsAggregator.buildAggregations(MetricsAggregator.java:41)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:180)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForAllBuckets(BucketsAggregator.java:224)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.bucket.geogrid.GeoGridAggregator.buildAggregations(GeoGridAggregator.java:160)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
	at org.elasticsearch.server@8.13.4/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
	at org.apache.lucene.core@9.10.0/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:117)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 6 more
"""
  },
  "status": 500
}

Hmm, does it happen if you set ignore_unmapped to true? I wonder if you've got some documents that don't have the required mappings? (see doc)?

Same result with ignore_unmapped set to true

I'm also trying the same query but on a different index with the same structure and its' giving me results but with failures in some shards, however, the number of shards where it fails varies. I don't know if that can point to something.
This error did not happen before, it started happening when we updated from version 8.5 to 8.13.4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.