# Error: CorruptIndexException when reading from gateway **URL:** \ **Category:** Elasticsearch\ **Created:** [June 1, 2010, 1:56pm UTC](https://discuss.elastic.co/t/error-corruptindexexception-when-reading-from-gateway/2987 "2010-06-01T13:56:12Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Yatir\_Ben\_Shlomo\_3](https://avatars.discourse-cdn.com/v4/letter/y/b38774/32.png) [@Yatir\_Ben\_Shlomo\_3](https://discuss.elastic.co/u/Yatir_Ben_Shlomo_3)\ **Post date:** [June 1, 2010, 1:56pm UTC](https://discuss.elastic.co/t/error-corruptindexexception-when-reading-from-gateway/2987/1 "2010-06-01T13:56:12Z")
see attached files: [http://elasticsearch-users.115913.n3.nabble.com/file/n861253/info.zip](http://elasticsearch-users.115913.n3.nabble.com/file/n861253/info.zip) Hi, I am testing the behavior of elastic search under large scale: in my setup there 2 64-bit nodes with 8 CPUs each. they running version 0.71 of ES (as a service), and using an index gateway in fs mode (NFS) I have a single index with 5 shards each shard has 2 replica (5/1) so I have 10 shards in total. I have indexed 8.5 Million documents here is the disk usage of all shards and transLog in the gateway 6.0G ./0/index 4.4M ./0/translog 6.2G ./1/index 2.3M ./1/translog 6.4G ./2/index 2.8M ./2/translog 6.9G ./3/index 1.4M ./3/translog 6.5G ./4/index 2.6M ./4/translog 32G . I then did the following things 1. stopped the indexing process 2. stopped one of the es nodes 3. waited about 3 minutes 4. stopped the other node 5. restarted the first node 6. queried for the number of docs: curl -XGET '[http://localhost:9200/en/\_count?q=](http://localhost:9200/en/_count?q=)_:_' I noticed the load on the machine was high (11-15, even now 30 minutes after the restart) at first I got zero results later after maybe 10 minutes I saw exceptions in the log ()see below) and I got only 6.5 M docs - one shard is corrupted I got a CorruptIndexException in the log file Attached also : [http://elasticsearch-users.115913.n3.nabble.com/file/n861253/info.zip](http://elasticsearch-users.115913.n3.nabble.com/file/n861253/info.zip) info.zip the cluster health, state info, nodes info and the full log file 07:23:33,508][WARN][indices.cluster] [Leeds, Betty Brant] Failed to start shard for index [en] and shard id [3] org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [en][3] Failed to perform recovery of translog at org.elasticsearch.index.gateway.fs.FsIndexShardGateway.recoverTranslog(FsIndexShardGateway.java: 381) at org.elasticsearch.index.gateway.fs.FsIndexShardGateway.recover(FsIndexShardGateway.java: 111) at org.elasticsearch.index.gateway.IndexShardGatewayService.recover(IndexShardGatewayService.java: 133) at org.elasticsearch.indices.cluster.IndicesClusterStateService $3.run(IndicesClusterStateService.java:342) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [en][3] Failed to open reader on writer at org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java: 166) at org.elasticsearch.index.shard.service.InternalIndexShard.performRecovery(InternalIndexShard.java: 407) at org.elasticsearch.index.gateway.fs.FsIndexShardGateway.recoverTranslog(FsIndexShardGateway.java: 378) ... 6 more Caused by: org.apache.lucene.index.CorruptIndexException: doc counts differ for segment \_1mfa: fieldsReader shows 334 but segmentInfo shows 3215 at org.apache.lucene.index.SegmentReader $CoreReaders.openDocStores(SegmentReader.java:282) at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:578) at org.apache.lucene.index.IndexWriter $ReaderPool.get(IndexWriter.java:609) at org.apache.lucene.index.IndexWriter $ReaderPool.getReadOnlyClone(IndexWriter.java:568) at org.apache.lucene.index.DirectoryReader.(DirectoryReader.java: 150) at org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java: 36) at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java: 405) at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java: 372) at org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java: 150) ... 8 more ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 1, 2010, 8:36pm UTC](https://discuss.elastic.co/t/error-corruptindexexception-when-reading-from-gateway/2987/2 "2010-06-01T20:36:37Z")
First, in elasticsearch lingo, 5 shards with 1 replica each equals 10. When a fresh node starts up, it will recover its state from the gateway. This means copying the relevant index files and transaction log for the relevant shards from it. In FS index storage with FS gateway, this means copying over the files, and applying the content of the transaction log. While this happens, if you query the cluster, you will get failures (check the shard failures in the response) saying the shards are not ready yet. This might take time, depending on your NFS, how big the index is and so on. I use native file copy when doing FS -\> FS, which should be the fastest way to do it, but still, it might take time. In order to know how long it took, what was spent on each state, and so on, you should set the 'index.gateway' logging to 'DEBUG' (right under 'action', same tabbing). Enable it to get a feeling how long these things take. The corrupt index probably got from not copying over the files properly. This might have to do with the NFS configuration. Can you post it? (clinton, if you are out there: can you share your NFS configuration?, it seems to work well for you). -shay.banon On Tue, Jun 1, 2010 at 4:56 PM, Yatir Ben Shlomo [yatirb@gmail.com](mailto:yatirb@gmail.com) wrote: > see attached files: > [http://elasticsearch-users.115913.n3.nabble.com/file/n861253/info.zip](http://elasticsearch-users.115913.n3.nabble.com/file/n861253/info.zip) > > Hi, > I am testing the behavior of Elasticsearch under large scale: > in my setup there 2 64-bit nodes with 8 CPUs each. > they running version 0.71 of ES (as a service), and using an index > gateway > in fs mode (NFS) > I have a single index with 5 shards each shard has 2 replica (5/1) so > I have > 10 shards in total. > I have indexed 8.5 Million documents > here is the disk usage of all shards and transLog in the gateway > 6.0G ./0/index > 4.4M ./0/translog > 6.2G ./1/index > 2.3M ./1/translog > 6.4G ./2/index > 2.8M ./2/translog > 6.9G ./3/index > 1.4M ./3/translog > 6.5G ./4/index > 2.6M ./4/translog > 32G . > > I then did the following things > > 1. stopped the indexing process > 2. stopped one of the es nodes > 3. waited about 3 minutes > 4. stopped the other node > 5. restarted the first node > 6. queried for the number of docs: curl -XGET > '[http://localhost:9200/en/\_count?q=\*:](http://localhost:9200/en/_count?q=*:)\*' > > I noticed the load on the machine was high (11-15, even now 30 minutes > after > the restart) > at first I got zero results > later after maybe 10 minutes I saw exceptions in the log ()see below) > and I got only 6.5 M docs - one shard is corrupted > > I got a CorruptIndexException in the log file > > Attached also : > [http://elasticsearch-users.115913.n3.nabble.com/file/n861253/info.zip](http://elasticsearch-users.115913.n3.nabble.com/file/n861253/info.zip) > info.zip the cluster health, state info, nodes info and the full log > file > > 07:23:33,508][WARN][indices.cluster] [Leeds, Betty Brant] > Failed to start shard for index [en] and shard id [3] > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > [en][3] Failed to perform recovery of translog > at > > org.elasticsearch.index.gateway.fs.FsIndexShardGateway.recoverTranslog(FsIndexShardGateway.java: > 381) > at > > org.elasticsearch.index.gateway.fs.FsIndexShardGateway.recover(FsIndexShardGateway.java: > 111) > at > > org.elasticsearch.index.gateway.IndexShardGatewayService.recover(IndexShardGatewayService.java: > 133) > at org.elasticsearch.indices.cluster.IndicesClusterStateService > $3.run(IndicesClusterStateService.java:342) > at java.util.concurrent.ThreadPoolExecutor > $Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > Caused by: > org.elasticsearch.index.engine.EngineCreationFailureException: [en][3] > Failed to open reader on writer > at > org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java: > 166) > at > > org.elasticsearch.index.shard.service.InternalIndexShard.performRecovery(InternalIndexShard.java: > 407) > at > > org.elasticsearch.index.gateway.fs.FsIndexShardGateway.recoverTranslog(FsIndexShardGateway.java: > 378) > ... 6 more > Caused by: org.apache.lucene.index.CorruptIndexException: doc counts > differ for segment \_1mfa: fieldsReader shows 334 but segmentInfo shows > 3215 > at org.apache.lucene.index.SegmentReader > $CoreReaders.openDocStores(SegmentReader.java:282) > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:578) > at org.apache.lucene.index.IndexWriter > $ReaderPool.get(IndexWriter.java:609) > at org.apache.lucene.index.IndexWriter > $ReaderPool.getReadOnlyClone(IndexWriter.java:568) > at > org.apache.lucene.index.DirectoryReader.(DirectoryReader.java: > 150) > at > > org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java: > 36) > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java: > 405) > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java: > 372) > at > org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java: > 150) > ... 8 more ---
**Author:** ![Yatir\_Ben\_Shlomo\_3](https://avatars.discourse-cdn.com/v4/letter/y/b38774/32.png) [@Yatir\_Ben\_Shlomo\_3](https://discuss.elastic.co/u/Yatir_Ben_Shlomo_3)\ **Post date:** [June 2, 2010, 9:11am UTC](https://discuss.elastic.co/t/error-corruptindexexception-when-reading-from-gateway/2987/3 "2010-06-02T09:11:24Z")
Hi Shay, Here is the NFS configuration: This is the mount command on the client: (from fstab) obnas:/obstore/gateway /outbrain/elasticsearch/gateway nfs rsize=8192,wsize=8192,timeo=14,intr Please tell me if you need more specific details thanks yatir On Jun 1, 11:36 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > First, in elasticsearch lingo, 5 shards with 1 replica each equals 10. > > When a fresh node starts up, it will recover its state from the gateway. > This means copying the relevant index files and transaction log for the > relevant shards from it. In FS index storage with FS gateway, this means > copying over the files, and applying the content of the transaction log. > While this happens, if you query the cluster, you will get failures (check > the shard failures in the response) saying the shards are not ready yet. > > This might take time, depending on your NFS, how big the index is and so on. > I use native file copy when doing FS -\> FS, which should be the fastest way > to do it, but still, it might take time. In order to know how long it took, > what was spent on each state, and so on, you should set the 'index.gateway' > logging to 'DEBUG' (right under 'action', same tabbing). Enable it to get a > feeling how long these things take. > > The corrupt index probably got from not copying over the files properly. > This might have to do with the NFS configuration. Can you post it? (clinton, > if you are out there: can you share your NFS configuration?, it seems to > work well for you). > > -shay.banon > > On Tue, Jun 1, 2010 at 4:56 PM, Yatir Ben Shlomo [yat...@gmail.com](mailto:yat...@gmail.com) wrote: > > > see attached files: > > [http://elasticsearch-users.115913.n3.nabble.com/file/n861253/info.zip](http://elasticsearch-users.115913.n3.nabble.com/file/n861253/info.zip) > > > Hi, > > I am testing the behavior of Elasticsearch under large scale: > > in my setup there 2 64-bit nodes with 8 CPUs each. > > they running version 0.71 of ES (as a service), and using an index > > gateway > > in fs mode (NFS) > > I have a single index with 5 shards each shard has 2 replica (5/1) so > > I have > > 10 shards in total. > > I have indexed 8.5 Million documents > > here is the disk usage of all shards and transLog in the gateway > > 6.0G ./0/index > > 4.4M ./0/translog > > 6.2G ./1/index > > 2.3M ./1/translog > > 6.4G ./2/index > > 2.8M ./2/translog > > 6.9G ./3/index > > 1.4M ./3/translog > > 6.5G ./4/index > > 2.6M ./4/translog > > 32G . > > > I then did the following things > > > > 1. stopped the indexing process > > 2. stopped one of the es nodes > > 3. waited about 3 minutes > > 4. stopped the other node > > 5. restarted the first node > > 6. queried for the number of docs: curl -XGET > > '[http://localhost:9200/en/\_count?q=\*:](http://localhost:9200/en/_count?q=*:)\*' > > > I noticed the load on the machine was high (11-15, even now 30 minutes > > after > > the restart) > > at first I got zero results > > later after maybe 10 minutes I saw exceptions in the log ()see below) > > and I got only 6.5 M docs - one shard is corrupted > > > I got a CorruptIndexException in the log file > > > Attached also : > > [http://elasticsearch-users.115913.n3.nabble.com/file/n861253/info.zip](http://elasticsearch-users.115913.n3.nabble.com/file/n861253/info.zip) > > info.zip the cluster health, state info, nodes info and the full log > > file > > > 07:23:33,508][WARN][indices.cluster] [Leeds, Betty Brant] > > Failed to start shard for index [en] and shard id [3] > > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > > [en][3] Failed to perform recovery of translog > > at > > > org.elasticsearch.index.gateway.fs.FsIndexShardGateway.recoverTranslog(FsIndexShardGateway.java: > > 381) > > at > > > org.elasticsearch.index.gateway.fs.FsIndexShardGateway.recover(FsIndexShardGateway.java: > > 111) > > at > > > org.elasticsearch.index.gateway.IndexShardGatewayService.recover(IndexShardGatewayService.java: > > 133) > > at org.elasticsearch.indices.cluster.IndicesClusterStateService > > $3.run(IndicesClusterStateService.java:342) > > at java.util.concurrent.ThreadPoolExecutor > > $Worker.runTask(ThreadPoolExecutor.java:886) > > at java.util.concurrent.ThreadPoolExecutor > > $Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:619) > > Caused by: > > org.elasticsearch.index.engine.EngineCreationFailureException: [en][3] > > Failed to open reader on writer > > at > > org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java: > > 166) > > at > > > org.elasticsearch.index.shard.service.InternalIndexShard.performRecovery(InternalIndexShard.java: > > 407) > > at > > > org.elasticsearch.index.gateway.fs.FsIndexShardGateway.recoverTranslog(FsIndexShardGateway.java: > > 378) > > ... 6 more > > Caused by: org.apache.lucene.index.CorruptIndexException: doc counts > > differ for segment \_1mfa: fieldsReader shows 334 but segmentInfo shows > > 3215 > > at org.apache.lucene.index.SegmentReader > > $CoreReaders.openDocStores(SegmentReader.java:282) > > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:578) > > at org.apache.lucene.index.IndexWriter > > $ReaderPool.get(IndexWriter.java:609) > > at org.apache.lucene.index.IndexWriter > > $ReaderPool.getReadOnlyClone(IndexWriter.java:568) > > at > > org.apache.lucene.index.DirectoryReader.(DirectoryReader.java: > > 150) > > at > > > org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java: > > 36) > > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java: > > 405) > > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java: > > 372) > > at > > org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java: > > 150) > > ... 8 more ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [June 2, 2010, 9:31am UTC](https://discuss.elastic.co/t/error-corruptindexexception-when-reading-from-gateway/2987/4 "2010-06-02T09:31:30Z")
Hiya > The corrupt index probably got from not copying over the files > properly. This might have to do with the NFS configuration. Can you > post it? (clinton, if you are out there: can you share your NFS > configuration?, it seems to work well for you). Actually, I'm using the default settings for mounting the NFS partition - nothing special there. However, the NFS server needs to perform reasonably well - you shouldn't use a cheap box for it. Especially during recovery or heavy indexing, a slow box can cause delays. Also, I'm setting some values in /etc/sysctl.conf which jgroups recommended - not sure if it is still relevant with zen: net.core.wmem\_max = 655360 net.core.rmem\_max = 26214400 And the only other thing I'm setting is the number of open files allowed: ulimit -n 20000 clint -- Web Announcements Limited is a company registered in England and Wales, with company number 05608868, with registered address at 10 Arvon Road, London, N5 1PR. ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [June 2, 2010, 9:33am UTC](https://discuss.elastic.co/t/error-corruptindexexception-when-reading-from-gateway/2987/5 "2010-06-02T09:33:47Z")
> Actually, I'm using the default settings for mounting the NFS partition > > - nothing special there. So in /etc/exports on the NFS server I have: /opt/es\_data 192.168.10.33(rw,root\_squash,sync,no\_subtree\_check) And in /etc/fstab on the client I have: data1:/opt/es\_data /opt/es\_data nfs rw 0 0 ## clint Web Announcements Limited is a company registered in England and Wales, with company number 05608868, with registered address at 10 Arvon Road, London, N5 1PR. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:23am UTC](https://discuss.elastic.co/t/error-corruptindexexception-when-reading-from-gateway/2987/6 "2017-07-06T04:23:37Z")