Resource not found exception in discovery

Hello,

We are starting to get lots of reports of random resource not found errors when searching in discovery. It looks like
the issue goes back to a version conflict for the search on elasticsearch. I am little bit of a loss on what to look at
next so help would be appreciated. The error and the relevant logs are below:

We are running kibana and elasticsearch 7.9.1 deployed via the kubernetes elastic-operator v1.3.0

Error
at Fetch._callee3$ (https://<HOST>/33912/bundles/core/core.entry.js:34:109213)
at l (https://<HOST>/33912/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:155323)
at Generator._invoke (https://<HOST>/33912/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:155076)
at Generator.forEach.e.<computed> [as next] (https://<HOST>/33912/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:155680)
at fetch_asyncGeneratorStep (https://<HOST>/33912/bundles/core/core.entry.js:34:102354)
at _next (https://<HOST>/33912/bundles/core/core.entry.js:34:102670)

kibana logs:

    {"type":"response","@timestamp":"2021-03-08T22:12:15Z","tags":[],"pid":7,"method":"delete","statusCode":200,"req":{"url":"/internal/search/es/FkNLY205V0U1U2tTa25id2lPQjlTcncgbEhFVzlLNmNUaFdmLVY1Mm55TS1pQToxNjY3ODMwMDg=","method":"delete","headers":{"connection":"upgrade","host":"<HOST>","x-forwarded-for":"172.218.4.190, 10.110.8.212","x-real-ip":"10.110.8.212","es-security-runas-user":"clio_admin","x-forwarded-proto":"https","x-forwarded-port":"443","x-amzn-trace-id":"Root=1-6046a13f-2a61b5c006e0f82271234856","sec-ch-ua":"\"Google Chrome\";v=\"89\", \"Chromium\";v=\"89\", \";Not A Brand\";v=\"99\"","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.72 Safari/537.36","kbn-version":"7.9.1","content-type":"application/json","accept":"*/*","origin":"<HOST>","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://<HOST>/app/discover","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"https://<HOST>/app/discover"},"res":{"statusCode":200,"responseTime":2001,"contentLength":9},"message":"DELETE /internal/search/es/FkNLY205V0U1U2tTa25id2lPQjlTcncgbEhFVzlLNmNUaFdmLVY1Mm55TS1pQToxNjY3ODMwMDg= 200 2001ms - 9.0B"}

Elasticsearch logs:

    {"type": "server", "timestamp": "2021-03-08T22:12:15,519Z", "level": "ERROR", "component": "o.e.x.c.a.AsyncResultsService", "cluster.name": "prod", "node.name": "prod-es-cold-001-az3-1", "message": "failed to update expiration time for async-search [FkNLY205V0U1U2tTa25id2lPQjlTcncgbEhFVzlLNmNUaFdmLVY1Mm55TS1pQToxNjY3ODMwMDg=]", "cluster.uuid": "bpxpeP_WRbaAbadGjsLn1g", "node.id": "lHEW9K6cThWf-V52nyM-iA" ,
      "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [prod-es-hot-az1-002-0][10.110.129.235:9300][indices:data/write/update[s]]",
      "Caused by: org.elasticsearch.index.engine.VersionConflictEngineException: [CKcm9WE5SkSknbwiOB9Srw]: version conflict, required seqNo [75712], primary term [10]. current document has seqNo [75720] and primary term [10]",
      "at org.elasticsearch.index.engine.InternalEngine.planIndexingAsPrimary(InternalEngine.java:1059) ~[elasticsearch-7.9.1.jar:7.9.1]",
      "at org.elasticsearch.index.engine.InternalEngine.indexingStrategyForOperation(InternalEngine.java:1022) ~[elasticsearch-7.9.1.jar:7.9.1]",
      "at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:911) ~[elasticsearch-7.9.1.jar:7.9.1]",
      "at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:819) ~[elasticsearch-7.9.1.jar:7.9.1]",
      "at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:791) ~[elasticsearch-7.9.1.jar:7.9.1]",
      "at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:748) ~[elasticsearch-7.9.1.jar:7.9.1]",
      "at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:274) ~[elasticsearch-7.9.1.jar:7.9.1]",
      "at org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:164) ~[elasticsearch-7.9.1.jar:7.9.1]",
      "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.1.jar:7.9.1]",
      "at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:209) ~[elasticsearch-7.9.1.jar:7.9.1]",
      "at org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:116) ~[elasticsearch-7.9.1.jar:7.9.1]",
      "at org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:82) ~[elasticsearch-7.9.1.jar:7.9.1]",
      "at org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:161) ~[elasticsearch-7.9.1.jar:7.9.1]",
      "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:710) ~[elasticsearch-7.9.1.jar:7.9.1]",
      "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.1.jar:7.9.1]",
      "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?]",
      "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?]",
      "at java.lang.Thread.run(Thread.java:832) [?:?]"] }

After digging around a bit more it seems likely to be this issue Failed to update expiration time for async-search · Issue #63213 · elastic/elasticsearch · GitHub and fixed by this issue Async search should retry updates on version conflict by jimczi · Pull Request #63652 · elastic/elasticsearch · GitHub included in elasticsearch 7.9.3 I will update in the next few days to confirm that fixes this.

Thank you for following up!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.