Facing java.io.EOFException: read past EOF exception and org.apache.lucene.index.CorruptIndexException: compound sub-files must have a valid codec header and footer: file is too small (0 bytes) in elastic 7.17.5

Elastic Stack Elasticsearch
1

In one our environment we are facing the "CorruptIndexException". While analyzing the elastic log we found the below are the list of exception details:

  1. infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | {"type": "server", "timestamp": "2023-06-30T11:18:47,132Z", "level": "INFO", "component": "o.e.i.g.GeoIpDownloader", "cluster.name": "docker-cluster", "node.name": "bd02cf096b77", "message": "updating geoip databases", "cluster.uuid": "V03V2wO-Thyovw0Vg_IQHg", "node.id": "hQMZMxJkTr6QsMmlqA2P2w" }

infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | {"type": "server", "timestamp": "2023-06-30T11:18:47,132Z", "level": "INFO", "component": "o.e.i.g.GeoIpDownloader", "cluster.name": "docker-cluster", "node.name": "bd02cf096b77", "message": "fetching geoip databases overview from [https://geoip.elastic.co/v1/database?elastic_geoip_service_tos=agree]", "cluster.uuid": "V03V2wO-Thyovw0Vg_IQHg", "node.id": "hQMZMxJkTr6QsMmlqA2P2w" }

infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | {"type": "server", "timestamp": "2023-06-30T11:18:57,181Z", "level": "ERROR", "component": "o.e.i.g.GeoIpDownloader", "cluster.name": "docker-cluster", "node.name": "bd02cf096b77", "message": "exception during geoip databases update", "cluster.uuid": "V03V2wO-Thyovw0Vg_IQHg", "node.id": "hQMZMxJkTr6QsMmlqA2P2w" ,
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "stacktrace": ["java.net.SocketTimeoutException: Connect timed out",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:543) ~[?:?]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:594) ~[?:?]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) ~[?:?]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at java.net.Socket.connect(Socket.java:633) ~[?:?]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:299) ~[?:?]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at sun.net.NetworkClient.doConnect(NetworkClient.java:178) ~[?:?]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at sun.net.www.http.HttpClient.openServer(HttpClient.java:498) ~[?:?]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at sun.net.www.http.HttpClient.openServer(HttpClient.java:603) ~[?:?]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at sun.net.www.protocol.https.HttpsClient.(HttpsClient.java:264) ~[?:?]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:378) ~[?:?]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:189) ~[?:?]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1242) ~[?:?]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1128) ~[?:?]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:175) ~[?:?]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1665) ~[?:?]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1589) ~[?:?]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:529) ~[?:?]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:308) ~[?:?]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.ingest.geoip.HttpClient.lambda$get$0(HttpClient.java:55) ~[ingest-geoip-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.ingest.geoip.HttpClient.doPrivileged(HttpClient.java:97) ~[ingest-geoip-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.ingest.geoip.HttpClient.get(HttpClient.java:49) ~[ingest-geoip-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.ingest.geoip.HttpClient.getBytes(HttpClient.java:40) ~[ingest-geoip-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.ingest.geoip.GeoIpDownloader.fetchDatabasesOverview(GeoIpDownloader.java:159) ~[ingest-geoip-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:147) ~[ingest-geoip-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:284) [ingest-geoip-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:100) [ingest-geoip-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:46) [ingest-geoip-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.persistent.NodePersistentTasksExecutor$1.doRun(NodePersistentTasksExecutor.java:42) [elasticsearch-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) [elasticsearch-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at java.lang.Thread.run(Thread.java:833) [?:?]"] }

This call is hapenning on every 3 days interval. is this caussed the CorruptIndexException?

  1. infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "stacktrace": ["org.apache.lucene.index.MergePolicy$MergeException: java.io.EOFException: read past EOF: MMapIndexInput(path="/usr/share/elasticsearch/data/nodes/0/indices/mRFo2vcbQxWkMhzFqZMTvg/1/index/_3e_Lucene80_0.dvd")",
    infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.index.engine.InternalEngine$EngineMergeScheduler$2.doRun(InternalEngine.java:2744) [elasticsearch-7.17.5.jar:7.17.5]",
    infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) [elasticsearch-7.17.5.jar:7.17.5]",
    infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.17.5.jar:7.17.5]",
    infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]",
    infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]",
    infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at java.lang.Thread.run(Thread.java:833) [?:?]",
    infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "Caused by: java.io.EOFException: read past EOF: MMapIndexInput(path="/usr/share/elasticsearch/data/nodes/0/indices/mRFo2vcbQxWkMhzFqZMTvg/1/index/_3e_Lucene80_0.dvd")",
    infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:85) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
    infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.store.DataInput.readInt(DataInput.java:102) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
    infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.store.ByteBufferIndexInput.readInt(ByteBufferIndexInput.java:174) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
    infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:194) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
    infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:255) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
    infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer.(Lucene80DocValuesProducer.java:99) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
    infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.codecs.lucene80.Lucene80DocValuesFormat.fieldsProducer(Lucene80DocValuesFormat.java:166) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
    infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.(PerFieldDocValuesFormat.java:303) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
    infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:386) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
    infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:51) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
    infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:67) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
    infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:175) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
    infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.index.SegmentReader.(SegmentReader.java:99) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
    infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:171) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
    infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.index.ReadersAndUpdates.getLatestReader(ReadersAndUpdates.java:235) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
    infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.index.PendingSoftDeletes.ensureInitialized(PendingSoftDeletes.java:189) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
    infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.index.PendingSoftDeletes.isFullyDeleted(PendingSoftDeletes.java:200) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
    infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.index.ReadersAndUpdates.isFullyDeleted(ReadersAndUpdates.java:748) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
    infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.index.IndexWriter.isFullyDeleted(IndexWriter.java:5505) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
    infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.index.IndexWriter.commitMerge(IndexWriter.java:4231) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
    infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4913) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
    infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4361) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
    infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:5920) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
    infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:626) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
    infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:94) ~[elasticsearch-7.17.5.jar:7.17.5]",
    infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:684) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]"] }

  2. infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | {"type": "server", "timestamp": "2023-07-12T23:04:16,430Z", "level": "WARN", "component": "o.e.i.c.IndicesClusterStateService", "cluster.name": "docker-cluster", "node.name": "bd02cf096b77", "message": "[passenger-history][1] marking and sending shard failed due to [shard failure, reason [refresh failed source[api]]]", "cluster.uuid": "V03V2wO-Thyovw0Vg_IQHg", "node.id": "hQMZMxJkTr6QsMmlqA2P2w" ,

infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "stacktrace": ["org.apache.lucene.index.CorruptIndexException: compound sub-files must have a valid codec header and footer: file is too small (0 bytes) (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="/usr/share/elasticsearch/data/nodes/0/indices/UAS5VDw1Sv6xrrrvoN39Bw/1/index/_52.kdm")))",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.codecs.CodecUtil.verifyAndCopyIndexHeader(CodecUtil.java:282) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.write(Lucene50CompoundFormat.java:93) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:47:22]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:5313) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.index.DocumentsWriterPerThread.sealFlushedSegment(DocumentsWriterPerThread.java:457) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:395) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:476) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:656) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:605) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:293) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:268) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:258) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:173) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:56) ~[elasticsearch-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:28) ~[elasticsearch-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:370) ~[elasticsearch-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:350) ~[elasticsearch-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:46:43]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1888) ~[elasticsearch-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1865) ~[elasticsearch-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:1285) ~[elasticsearch-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.action.admin.indices.refresh.TransportShardRefreshAction.lambda$shardOperationOnPrimary$0(TransportShardRefreshAction.java:72) ~[elasticsearch-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:436) ~[elasticsearch-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:71) ~[elasticsearch-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:28) ~[elasticsearch-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1153) ~[elasticsearch-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:124) ~[elasticsearch-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:508) ~[elasticsearch-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:414) ~[elasticsearch-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$23(IndexShard.java:3450) ~[elasticsearch-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:219) ~[elasticsearch-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:253) ~[elasticsearch-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:199) ~[elasticsearch-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:3421) ~[elasticsearch-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:1090) ~[elasticsearch-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:411) ~[elasticsearch-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.action.support.replication.TransportReplicationAction.handlePrimaryRequest(TransportReplicationAction.java:355) ~[elasticsearch-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:341) ~[?:?]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:417) ~[?:?]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:67) ~[elasticsearch-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:1045) ~[elasticsearch-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) ~[elasticsearch-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.17.5.jar:7.17.5]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]",
infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | "at java.lang.Thread.run(Thread.java:833) [?:?]"] }

infinity_infinity-elasticsearch.1.862455ajz1ca@WorkerNode03Prod | {"type": "server", "timestamp": "2023-07-12T23:04:16,439Z", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "docker-cluster", "node.name": "bd02cf096b77", "message": "Cluster health status changed from [YELLOW] to [RED] (reason: [shards failed [[passenger-history][1]]]).", "cluster.uuid": "V03V2wO-Thyovw0Vg_IQHg", "node.id": "hQMZMxJkTr6QsMmlqA2P2w" }

Above 3 error message only we are seeing in the elastic log.

For storage we are using VM local disk[Gluster file system(glusterfs 7.2)], for deployment we are using docker-swarm.

We are currently using Elasticsearch version 7.17.5 and spring-data-elasticsearch version 4.4.2.

Has anyone else encountered a similar issue?

2

Failure to downloadthe geoip file is unlrelated to the corruption issue. I suspect that your issue is related to the type of storage you are using. See this recent thread for more details.

3

Thank you @Christian_Dahlqvist . Will have a look on response you have shared.

4

Copying the conclusion from the other thread here to avoid it getting lost in any further conversation:

GlusterFS isn't at all a local disk and the error you're seeing indicates it does not behave like a local disk accurately enough for Elasticsearch. See these docs for more information:

Elasticsearch requires the filesystem to act as if it were backed by a local disk, but this means that it will work correctly on properly-configured remote block devices (e.g. a SAN) and remote filesystems (e.g. NFS) as long as the remote storage behaves no differently from local storage.

5

Thank you @DavidTurner . Will check with our dev ops team on this.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.