# SOLVED - ELASTICSEARCH - Unable to start elastic search service on linux **URL:** \ **Category:** Elasticsearch\ **Created:** [March 29, 2016, 8:45pm UTC](https://discuss.elastic.co/t/solved-elasticsearch-unable-to-start-elastic-search-service-on-linux/45732 "2016-03-29T20:45:38Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![vbondalapati](https://avatars.discourse-cdn.com/v4/letter/v/e274bd/32.png) [@vbondalapati](https://discuss.elastic.co/u/vbondalapati)\ **Post date:** [March 29, 2016, 8:45pm UTC](https://discuss.elastic.co/t/solved-elasticsearch-unable-to-start-elastic-search-service-on-linux/45732/1 "2016-03-29T20:45:38Z")
It all started when we terminated the the over-running snapshot restore process. after this when we tried to start the elastic search we are getting below error . could some please help , its a elastic search 1.3.4 version on linux We are not in a position to start the fresh install - so any small hacks to get around the problem would be helpful. Details from log - some data masked [2016-03-29 08:24:12,543][WARN][cluster.action.shard] [instance\_9300] [default-index][3] received shard failed for [default-index][3], node[XXXXXXXXX], [P], s[INITIALIZING], indexUUID [nR\_XXXXXXX], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[default-index][3] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[default-index][3] shard allocated for local recovery (post api), should exist, but doesn't, current files: [\_222.si, \_2222.fdx, 1111.fnm, ----- list of all files]]; nested: FileNotFoundException[No such file [\_3yxri.si]]; ]] ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 29, 2016, 8:47pm UTC](https://discuss.elastic.co/t/solved-elasticsearch-unable-to-start-elastic-search-service-on-linux/45732/2 "2016-03-29T20:47:47Z")
Based on that log it looks like ES has started. Can you not curl the host on 9200? ---
**Author:** ![vbondalapati](https://avatars.discourse-cdn.com/v4/letter/v/e274bd/32.png) [@vbondalapati](https://discuss.elastic.co/u/vbondalapati)\ **Post date:** [March 29, 2016, 9:13pm UTC](https://discuss.elastic.co/t/solved-elasticsearch-unable-to-start-elastic-search-service-on-linux/45732/3 "2016-03-29T21:13:03Z")
Mark - thanks for your response. I can do curl on the host and getting a response - but the shards are getting MARKED as UNASSIGNED . shard 3 is alone complaining and remaining shards looks OK. shard 3 is distributed on host 3 & host 4 which are continuously doing excessive logging and so we are keeping them down our set-up is as below 1. total 4 hosts having each ES node 2. 5 shards $curl --fail -XGET '[http://localhost:9200/\_cat/shards?pretty=true](http://localhost:9200/_cat/shards?pretty=true)' index 4 p STARTED 4336176 14.8gb IP1 host1\_9300 index 4 r STARTED 4336176 14.8gb IP4 host4\_9300 index 0 p STARTED 4336256 14gb IP1 host1\_9300 index 0 r STARTED 4336256 14gb IP4 host4\_9300 index 3 p INITIALIZING IP3 host3\_9300 index 3 r UNASSIGNED index 1 r STARTED 4340540 14.5gb IP2 host2\_9300 index 1 p STARTED 4340540 14.5gb IP1 host1\_9300 index 2 r STARTED 4333466 15.1gb IP2 host2\_9300 index 2 p STARTED 4333466 15.1gb IP4 host4\_9300 ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 29, 2016, 9:24pm UTC](https://discuss.elastic.co/t/solved-elasticsearch-unable-to-start-elastic-search-service-on-linux/45732/4 "2016-03-29T21:24:04Z")
Then ES has started 🙂 You will probably need to look at the logs on `host3` and see what is happening. ---
**Author:** ![vbondalapati](https://avatars.discourse-cdn.com/v4/letter/v/e274bd/32.png) [@vbondalapati](https://discuss.elastic.co/u/vbondalapati)\ **Post date:** [March 29, 2016, 10:44pm UTC](https://discuss.elastic.co/t/solved-elasticsearch-unable-to-start-elastic-search-service-on-linux/45732/5 "2016-03-29T22:44:41Z")
Hi mark- the log i have attached in the description is the content of log from host3 and it is complaining abt file not found exception [2016-03-29 08:24:12,543][WARN][cluster.action.shard] [instance\_9300] [default-index][3] received shard failed for [default-index][3], node[XXXXXXXXX], [P], s[INITIALIZING], indexUUID [nR\_XXXXXXX], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[default-index][3] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[default-index][3] shard allocated for local recovery (post api), should exist, but doesn't, current files: [\_222.si, \_2222.fdx, 1111.fnm, ----- list of all files]]; nested: FileNotFoundException[No such file [\_3yxri.si]]; ]] ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 30, 2016, 1:42am UTC](https://discuss.elastic.co/t/solved-elasticsearch-unable-to-start-elastic-search-service-on-linux/45732/6 "2016-03-30T01:42:38Z")
Ouch, you might need to shutdown that node so that the replica can take its place. Then - **upgrade**! Versions prior to 1.5 have known corruption issues. ---
**Author:** ![vbondalapati](https://avatars.discourse-cdn.com/v4/letter/v/e274bd/32.png) [@vbondalapati](https://discuss.elastic.co/u/vbondalapati)\ **Post date:** [March 30, 2016, 8:39am UTC](https://discuss.elastic.co/t/solved-elasticsearch-unable-to-start-elastic-search-service-on-linux/45732/7 "2016-03-30T08:39:42Z")
HI Mark - our situation is the corruption is on Disaster recovery box and primary elasticsearch modules are still up and running . Solution we are looking is to fix the current issue and resolve the UNASSIGNED shards . Elastic search upgrade is planned in future. Any help to resolve the UNASSIGNED shards issue would be great . ---
**Author:** ![vbondalapati](https://avatars.discourse-cdn.com/v4/letter/v/e274bd/32.png) [@vbondalapati](https://discuss.elastic.co/u/vbondalapati)\ **Post date:** [March 30, 2016, 2:51pm UTC](https://discuss.elastic.co/t/solved-elasticsearch-unable-to-start-elastic-search-service-on-linux/45732/8 "2016-03-30T14:51:29Z")
ANy one can suggest a solution for the issue .. we need to get around the UNASSIGNED SHARD issue ---
**Author:** ![vbondalapati](https://avatars.discourse-cdn.com/v4/letter/v/e274bd/32.png) [@vbondalapati](https://discuss.elastic.co/u/vbondalapati)\ **Post date:** [March 31, 2016, 9:10pm UTC](https://discuss.elastic.co/t/solved-elasticsearch-unable-to-start-elastic-search-service-on-linux/45732/9 "2016-03-31T21:10:36Z")
we are having one of the shard corrupted - is there any way to re-create the shard alone . $java -cp elasticsearch-1.3.4/lib/lucene-core-4.9.1.jar -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex $shard\_d -verbose Opening index @ /data/host\_9300/es\_nameXXX/nodes/0/indices/default-index/3/index/ ERROR: could not read any segments file in directory java.nio.file.NoSuchFileException: /data/host\_9300/es\_nameXXX/nodes/0/indices/default-index/3/index/\_3XXXX.si at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177) at java.nio.channels.FileChannel.open(FileChannel.java:287) at java.nio.channels.FileChannel.open(FileChannel.java:335) at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:196) at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:113) at org.apache.lucene.codecs.lucene46.Lucene46SegmentInfoReader.read(Lucene46SegmentInfoReader.java:49) at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:361) at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:457) at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:912) at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:758) at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:453) at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:398) at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:2051) ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 31, 2016, 9:41pm UTC](https://discuss.elastic.co/t/solved-elasticsearch-unable-to-start-elastic-search-service-on-linux/45732/10 "2016-03-31T21:41:38Z")
Check the other node that has the replica, does that actually have data in the shard directory? ---
**Author:** ![vbondalapati](https://avatars.discourse-cdn.com/v4/letter/v/e274bd/32.png) [@vbondalapati](https://discuss.elastic.co/u/vbondalapati)\ **Post date:** [April 1, 2016, 9:19am UTC](https://discuss.elastic.co/t/solved-elasticsearch-unable-to-start-elastic-search-service-on-linux/45732/11 "2016-04-01T09:19:05Z")
> [@vbondalapati](#): > > si We checked and data it is complaining and is not available in both primary & replica shards - looks like we lost some data while doing sync between prod & DR . we are OK to have some data loss and get the shard started some how and reason for that is as below 1. shard corruption is on Disaster recovery box and once the shard changes from UNASSIGNED -STARTED we will sync the snapshots from prod to dr box and the do a restore. ---
**Author:** ![vbondalapati](https://avatars.discourse-cdn.com/v4/letter/v/e274bd/32.png) [@vbondalapati](https://discuss.elastic.co/u/vbondalapati)\ **Post date:** [April 1, 2016, 4:07pm UTC](https://discuss.elastic.co/t/solved-elasticsearch-unable-to-start-elastic-search-service-on-linux/45732/12 "2016-04-01T16:07:27Z")
I managed to the fix the corrupted shard issue by deleting the shard alone by re-route all data in the shard directory for the index is lost and snapshot is restored from primary. curl -XPOST 'localhost:9200/\_cluster/reroute' -d '{"commands": [{"allocate": {"index": "", "shard": 3, "node": "node\_name", "allow\_primary": true }}]}' can have a peaceful weekend 😌 some commands if you need for troubleshooting. curl '[http://localhost:9200/\_cat/segments?v](http://localhost:9200/_cat/segments?v)' curl -XGET '[http://localhost:9200/\_recovery?pretty=true](http://localhost:9200/_recovery?pretty=true)' curl -XGET '[http://localhost:9200/index\_name/\_recovery?pretty=true](http://localhost:9200/index_name/_recovery?pretty=true)' curl -XGET '[http://localhost:9200/\_cluster/health?level=indices&pretty](http://localhost:9200/_cluster/health?level=indices&pretty)' curl -XGET 'localhost:9200/\_cat/recovery?v' curl -XGET '[http://localhost:9200/\_cat/shards?pretty=true](http://localhost:9200/_cat/shards?pretty=true)' curl -XGET '[http://localhost:9200/\_cluster/health?pretty=true](http://localhost:9200/_cluster/health?pretty=true)' ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [April 1, 2016, 8:46pm UTC](https://discuss.elastic.co/t/solved-elasticsearch-unable-to-start-elastic-search-service-on-linux/45732/13 "2016-04-01T20:46:23Z")
That's it really fixing it as you lost the data. I'd strongly recommend upgrading as previously mentioned. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:03pm UTC](https://discuss.elastic.co/t/solved-elasticsearch-unable-to-start-elastic-search-service-on-linux/45732/14 "2017-07-05T23:03:06Z")