Hello,
We are starting to get lots of reports of random resource not found errors when searching in discovery. It looks like
the issue goes back to a version conflict for the search on elasticsearch. I am little bit of a loss on what to look at
next so help would be appreciated. The error and the relevant logs are below:
We are running kibana and elasticsearch 7.9.1 deployed via the kubernetes elastic-operator v1.3.0
Error
at Fetch._callee3$ (https://<HOST>/33912/bundles/core/core.entry.js:34:109213)
at l (https://<HOST>/33912/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:155323)
at Generator._invoke (https://<HOST>/33912/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:155076)
at Generator.forEach.e.<computed> [as next] (https://<HOST>/33912/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:155680)
at fetch_asyncGeneratorStep (https://<HOST>/33912/bundles/core/core.entry.js:34:102354)
at _next (https://<HOST>/33912/bundles/core/core.entry.js:34:102670)
kibana logs:
{"type":"response","@timestamp":"2021-03-08T22:12:15Z","tags":[],"pid":7,"method":"delete","statusCode":200,"req":{"url":"/internal/search/es/FkNLY205V0U1U2tTa25id2lPQjlTcncgbEhFVzlLNmNUaFdmLVY1Mm55TS1pQToxNjY3ODMwMDg=","method":"delete","headers":{"connection":"upgrade","host":"<HOST>","x-forwarded-for":"172.218.4.190, 10.110.8.212","x-real-ip":"10.110.8.212","es-security-runas-user":"clio_admin","x-forwarded-proto":"https","x-forwarded-port":"443","x-amzn-trace-id":"Root=1-6046a13f-2a61b5c006e0f82271234856","sec-ch-ua":"\"Google Chrome\";v=\"89\", \"Chromium\";v=\"89\", \";Not A Brand\";v=\"99\"","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.72 Safari/537.36","kbn-version":"7.9.1","content-type":"application/json","accept":"*/*","origin":"<HOST>","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://<HOST>/app/discover","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"https://<HOST>/app/discover"},"res":{"statusCode":200,"responseTime":2001,"contentLength":9},"message":"DELETE /internal/search/es/FkNLY205V0U1U2tTa25id2lPQjlTcncgbEhFVzlLNmNUaFdmLVY1Mm55TS1pQToxNjY3ODMwMDg= 200 2001ms - 9.0B"}
Elasticsearch logs:
{"type": "server", "timestamp": "2021-03-08T22:12:15,519Z", "level": "ERROR", "component": "o.e.x.c.a.AsyncResultsService", "cluster.name": "prod", "node.name": "prod-es-cold-001-az3-1", "message": "failed to update expiration time for async-search [FkNLY205V0U1U2tTa25id2lPQjlTcncgbEhFVzlLNmNUaFdmLVY1Mm55TS1pQToxNjY3ODMwMDg=]", "cluster.uuid": "bpxpeP_WRbaAbadGjsLn1g", "node.id": "lHEW9K6cThWf-V52nyM-iA" ,
"stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [prod-es-hot-az1-002-0][10.110.129.235:9300][indices:data/write/update[s]]",
"Caused by: org.elasticsearch.index.engine.VersionConflictEngineException: [CKcm9WE5SkSknbwiOB9Srw]: version conflict, required seqNo [75712], primary term [10]. current document has seqNo [75720] and primary term [10]",
"at org.elasticsearch.index.engine.InternalEngine.planIndexingAsPrimary(InternalEngine.java:1059) ~[elasticsearch-7.9.1.jar:7.9.1]",
"at org.elasticsearch.index.engine.InternalEngine.indexingStrategyForOperation(InternalEngine.java:1022) ~[elasticsearch-7.9.1.jar:7.9.1]",
"at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:911) ~[elasticsearch-7.9.1.jar:7.9.1]",
"at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:819) ~[elasticsearch-7.9.1.jar:7.9.1]",
"at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:791) ~[elasticsearch-7.9.1.jar:7.9.1]",
"at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:748) ~[elasticsearch-7.9.1.jar:7.9.1]",
"at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:274) ~[elasticsearch-7.9.1.jar:7.9.1]",
"at org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:164) ~[elasticsearch-7.9.1.jar:7.9.1]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.1.jar:7.9.1]",
"at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:209) ~[elasticsearch-7.9.1.jar:7.9.1]",
"at org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:116) ~[elasticsearch-7.9.1.jar:7.9.1]",
"at org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:82) ~[elasticsearch-7.9.1.jar:7.9.1]",
"at org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:161) ~[elasticsearch-7.9.1.jar:7.9.1]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:710) ~[elasticsearch-7.9.1.jar:7.9.1]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.1.jar:7.9.1]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?]",
"at java.lang.Thread.run(Thread.java:832) [?:?]"] }