# "Failed to merge" - java.io.FileNotFoundException **URL:** \ **Category:** Elasticsearch\ **Created:** [January 12, 2012, 6:33pm UTC](https://discuss.elastic.co/t/failed-to-merge-java-io-filenotfoundexception/6367 "2012-01-12T18:33:05Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Frederic](https://avatars.discourse-cdn.com/v4/letter/f/4491bb/32.png) [@Frederic](https://discuss.elastic.co/u/Frederic)\ **Post date:** [January 12, 2012, 6:33pm UTC](https://discuss.elastic.co/t/failed-to-merge-java-io-filenotfoundexception/6367/1 "2012-01-12T18:33:05Z")
Hi guys, I've been indexing millions of docs in a ES cluster with 4 processes that lanches about 10 threads each, and each one of those threads use a Transport client for indexing. For one day everything was fine, but today we started to experience lots of slow queries to ES and found the following output in the logs of almost every server (there are 6 ES servers): [2012-01-12 00:20:38,510][WARN][index.merge.scheduler] [Phage] [items][14] failed to merge java.io.FileNotFoundException: \_b7i6\_1.del at org.elasticsearch.index.store.Store $StoreDirectory.fileLength(Store.java:378) at org.apache.lucene.index.SegmentInfo.sizeInBytes(SegmentInfo.java: 303) at org.apache.lucene.index.MergePolicy $OneMerge.totalBytesSize(MergePolicy.java:174) at org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java: 79) at org.apache.lucene.index.ConcurrentMergeScheduler $MergeThread.run(ConcurrentMergeScheduler.java:456) I've 'gisted' the cluster state here [https://gist.github.com/1602222](https://gist.github.com/1602222) What could be the reasons for this warning? The cluster health shows everything ok so, I guess no information has been lost, but since these are production servers, I'd rather know if this can be a problem as soon as possible. Thanks in advance! Frederic ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [January 12, 2012, 9:52pm UTC](https://discuss.elastic.co/t/failed-to-merge-java-io-filenotfoundexception/6367/2 "2012-01-12T21:52:19Z")
Did someone delete some files from the nodes? Were there any failures before it started to happen? Which version are you using? On Thu, Jan 12, 2012 at 8:33 PM, Frederic [focampo.br@gmail.com](mailto:focampo.br@gmail.com) wrote: > Hi guys, > > I've been indexing millions of docs in a ES cluster with 4 processes > that lanches about 10 threads each, and each one of those threads use > a Transport client for indexing. > > For one day everything was fine, but today we started to experience > lots of slow queries to ES and found the following output in the logs > of almost every server (there are 6 ES servers): > > [2012-01-12 00:20:38,510][WARN][index.merge.scheduler] [Phage] > [items][14] failed to merge > java.io.FileNotFoundException: \_b7i6\_1.del > at org.elasticsearch.index.store.Store > $StoreDirectory.fileLength(Store.java:378) > at org.apache.lucene.index.SegmentInfo.sizeInBytes(SegmentInfo.java: > 303) > at org.apache.lucene.index.MergePolicy > $OneMerge.totalBytesSize(MergePolicy.java:174) > at > > org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java: > 79) > at org.apache.lucene.index.ConcurrentMergeScheduler > $MergeThread.run(ConcurrentMergeScheduler.java:456) > > I've 'gisted' the cluster state here [ES Stats and Health · GitHub](https://gist.github.com/1602222) > > What could be the reasons for this warning? > The cluster health shows everything ok so, I guess no information has > been lost, but since these are production servers, I'd rather know if > this can be a problem as soon as possible. > > Thanks in advance! > > Frederic ---
**Author:** ![Frederic](https://avatars.discourse-cdn.com/v4/letter/f/4491bb/32.png) [@Frederic](https://discuss.elastic.co/u/Frederic)\ **Post date:** [January 13, 2012, 1:48pm UTC](https://discuss.elastic.co/t/failed-to-merge-java-io-filenotfoundexception/6367/3 "2012-01-13T13:48:10Z")
Thanks for you reply and sorry for the missing info: - I'm using ES 0.18.5 - AFAIK nobody should have deleted any file from those servers - In some servers, only the mentioned output has been logged so far (last night it was logged another similar warning, but with a different file name java.io.FileNotFoundException: \_cqjh\_2.del''). In some other servers there are only DEBUG stacktraces like the following [2012-01-10 15:08:59,178][DEBUG][action.admin.cluster.node.info] [James Howlett] failed to execute on node [pfh3QO6ATo2A4wcEk2Cq0g] org.elasticsearch.transport.RemoteTransportException: [Gog][inet[/172.16.138.113:9300]][/cluster/nodes/info/node] Caused by: java.lang.NullPointerException at org.elasticsearch.http.HttpInfo.writeTo(HttpInfo.java:65) I assumed this as expected as I've removed a server from the cluster, whose IP was configured in the 'discovery.zen.ping.unicast.hosts' list. I'm planning to restart each one of the ES instances because some other issues (too many open idle connections), could that possibly recreate the missing files? Thanks a lot, On 12 ene, 18:52, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > Did someone delete some files from the nodes? Were there any failures > before it started to happen? Which version are you using? > > On Thu, Jan 12, 2012 at 8:33 PM, Frederic [focampo...@gmail.com](mailto:focampo...@gmail.com) wrote: > > > Hi guys, > > > I've been indexing millions of docs in a ES cluster with 4 processes > > that lanches about 10 threads each, and each one of those threads use > > a Transport client for indexing. > > > For one day everything was fine, but today we started to experience > > lots of slow queries to ES and found the following output in the logs > > of almost every server (there are 6 ES servers): > > > [2012-01-12 00:20:38,510][WARN][index.merge.scheduler] [Phage] > > [items][14] failed to merge > > java.io.FileNotFoundException: \_b7i6\_1.del > > at org.elasticsearch.index.store.Store > > $StoreDirectory.fileLength(Store.java:378) > > at org.apache.lucene.index.SegmentInfo.sizeInBytes(SegmentInfo.java: > > 303) > > at org.apache.lucene.index.MergePolicy > > $OneMerge.totalBytesSize(MergePolicy.java:174) > > at > > > org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java: > > 79) > > at org.apache.lucene.index.ConcurrentMergeScheduler > > $MergeThread.run(ConcurrentMergeScheduler.java:456) > > > I've 'gisted' the cluster state herehttps://gist.github.com/1602222 > > > What could be the reasons for this warning? > > The cluster health shows everything ok so, I guess no information has > > been lost, but since these are production servers, I'd rather know if > > this can be a problem as soon as possible. > > > Thanks in advance! > > > Frederic ---
**Author:** ![T\_Vinod\_Gupta](https://avatars.discourse-cdn.com/v4/letter/t/fbc32d/32.png) [@T\_Vinod\_Gupta](https://discuss.elastic.co/u/T_Vinod_Gupta)\ **Post date:** [July 9, 2012, 5:51am UTC](https://discuss.elastic.co/t/failed-to-merge-java-io-filenotfoundexception/6367/4 "2012-07-09T05:51:07Z")
Were you able to resolve your issue? if yes, how? im facing a similar issue and my queries are getting really slow. not sure how to fix it. thanks On Thursday, January 12, 2012 10:33:05 AM UTC-8, Frederic wrote: > Hi guys, > > I've been indexing millions of docs in a ES cluster with 4 processes > that lanches about 10 threads each, and each one of those threads use > a Transport client for indexing. > > For one day everything was fine, but today we started to experience > lots of slow queries to ES and found the following output in the logs > of almost every server (there are 6 ES servers): > > [2012-01-12 00:20:38,510][WARN][index.merge.scheduler] [Phage] > [items][14] failed to merge > java.io.FileNotFoundException: \_b7i6\_1.del > at org.elasticsearch.index.store.Store > $StoreDirectory.fileLength(Store.java:378) > at org.apache.lucene.index.SegmentInfo.sizeInBytes(SegmentInfo.java: > 303) > at org.apache.lucene.index.MergePolicy > $OneMerge.totalBytesSize(MergePolicy.java:174) > at > org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java: > > 1. > > at org.apache.lucene.index.ConcurrentMergeScheduler > $MergeThread.run(ConcurrentMergeScheduler.java:456) > > I've 'gisted' the cluster state here [ES Stats and Health · GitHub](https://gist.github.com/1602222) > > What could be the reasons for this warning? > The cluster health shows everything ok so, I guess no information has > been lost, but since these are production servers, I'd rather know if > this can be a problem as soon as possible. > > Thanks in advance! > > Frederic ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:21am UTC](https://discuss.elastic.co/t/failed-to-merge-java-io-filenotfoundexception/6367/5 "2017-07-06T03:21:03Z")