Hi All,
This is my first post, I was going through my elasticsearch logs and found the frequent SocketTimeoutException (stack trace posted below). We are currently using elasticsearch v2.3.3, we have 3 master nodes behind an AWS ELB and 20 Data nodes behind an AWS ELB. We have a Java application, which reads through the kafka queue and pushes them to Elasticsearch and we observe this error frequently. Any clue why it is failing.
2017-05-31 16:30:27,940][WARN ][index.snapshots.blobstore] [ElasticSearch_stage_134-92-243-59] failed to read index file [index-178]
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
at sun.security.ssl.InputRecord.read(InputRecord.java:480)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:934)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:891)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:198)
at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:178)
at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137)
at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:151)
at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:151)
at java.security.DigestInputStream.read(DigestInputStream.java:161)
at com.amazonaws.services.s3.internal.DigestValidationInputStream.read(DigestValidationInputStream.java:59)
at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at com.google.common.io.ByteStreams.copy(ByteStreams.java:70)
at com.google.common.io.ByteStreams.toByteArray(ByteStreams.java:115)
at org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.readBlob(ChecksumBlobStoreFormat.java:94)
at org.elasticsearch.repositories.blobstore.BlobStoreFormat.read(BlobStoreFormat.java:83)
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$Context.buildBlobStoreIndexShardSnapshots(BlobStoreIndexShardRepository.java:469)
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$SnapshotContext.snapshot(BlobStoreIndexShardRepository.java:537)
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository.snapshot(BlobStoreIndexShardRepository.java:191)
at org.elasticsearch.snapshots.SnapshotShardsService.snapshot(SnapshotShardsService.java:340)
at org.elasticsearch.snapshots.SnapshotShardsService.access$200(SnapshotShardsService.java:76)
at org.elasticsearch.snapshots.SnapshotShardsService$1.doRun(SnapshotShardsService.java:296)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Thanks
Manoj Badam