# Forcemerge failed ( java.lang.ArrayIndexOutOfBoundsException ) **URL:** \ **Category:** Elasticsearch\ **Created:** [June 18, 2018, 2:05pm UTC](https://discuss.elastic.co/t/forcemerge-failed-java-lang-arrayindexoutofboundsexception/136317 "2018-06-18T14:05:39Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![pfseth](https://avatars.discourse-cdn.com/v4/letter/p/e9bcb4/32.png) [@pfseth](https://discuss.elastic.co/u/pfseth)\ **Post date:** [June 18, 2018, 2:05pm UTC](https://discuss.elastic.co/t/forcemerge-failed-java-lang-arrayindexoutofboundsexception/136317/1 "2018-06-18T14:05:39Z")
Hi, I run curator ( forcemerge ) on previous day's indices to reduce the segment sizes, however of late i have observed the merge operation failing and in turn brings down the cluster. Appreciate any feedback ( Image attached ) Elasticsearch Version : 6.2.2 ![1](https://us1.discourse-cdn.com/elastic/original/3X/b/a/ba8d112ebdf9f15f3ae1c020eeca112ad7e6c056.png) ![2](https://us1.discourse-cdn.com/elastic/original/3X/c/b/cb16d78a13a881aa73225ac11c1f37a13ee8abb7.png) ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [June 18, 2018, 2:18pm UTC](https://discuss.elastic.co/t/forcemerge-failed-java-lang-arrayindexoutofboundsexception/136317/2 "2018-06-18T14:18:48Z")
Please don't post images of text as they are hardly readable and not searchable. Instead paste the text and format it with `` icon. Check the preview window. ---
**Author:** ![pfseth](https://avatars.discourse-cdn.com/v4/letter/p/e9bcb4/32.png) [@pfseth](https://discuss.elastic.co/u/pfseth)\ **Post date:** [June 18, 2018, 2:27pm UTC](https://discuss.elastic.co/t/forcemerge-failed-java-lang-arrayindexoutofboundsexception/136317/3 "2018-06-18T14:27:45Z")
Well tried that earlier, the limit is set to 7000 characters, hence posted the image. Is there the option to upload log files. Truncated logs ... ``` [2018-06-17T03:20:58,080][WARN][o.e.c.a.s.ShardStateAction] [master] [cluster-metricbeat-2018.06.17][0] received shard failed for shard id [[cluster-metricbeat-2018.06.17][0]], allocation id [6mNTb2ZpThOzkm9tqR zFQQ], primary term [0], message [shard failure, reason [merge failed]], failure [NotSerializableExceptionWrapper[merge_exception: java.lang.ArrayIndexOutOfBoundsException: 23]; nested: ArrayIndexOutOfBoundsExce ption[23]; ] org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: merge_exception: java.lang.ArrayIndexOutOfBoundsException: 23 at org.elasticsearch.index.engine.InternalEngine$EngineMergeScheduler$2.doRun(InternalEngine.java:2101) ~[elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.2.jar:6.2.2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_161] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_161] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161] Caused by: java.lang.ArrayIndexOutOfBoundsException: 23 at org.apache.lucene.codecs.lucene70.Lucene70DocValuesProducer$7.longValue(Lucene70DocValuesProducer.java:552) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43] at org.apache.lucene.index.SingletonSortedNumericDocValues.nextValue(SingletonSortedNumericDocValues.java:73) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43] at org.apache.lucene.codecs.DocValuesConsumer$3$1.nextValue(DocValuesConsumer.java:446) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43] at org.apache.lucene.codecs.lucene70.Lucene70DocValuesConsumer.writeValues(Lucene70DocValuesConsumer.java:162) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43] at org.apache.lucene.codecs.lucene70.Lucene70DocValuesConsumer.addSortedNumericField(Lucene70DocValuesConsumer.java:530) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43] at org.apache.lucene.codecs.DocValuesConsumer.mergeSortedNumericField(DocValuesConsumer.java:371) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43] at org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:143) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43] at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.merge(PerFieldDocValuesFormat.java:151) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01 -10 00:48:43] at org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:181) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43] at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:125) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43] at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4443) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43] at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4083) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43] at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43] at org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:99) ~[elasticsearch-6.2.2.jar:6.2.2] at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:4 3] [2018-06-17T03:20:58,161][INFO][o.e.c.r.a.AllocationService] [master] Cluster health status changed from [GREEN] to [YELLOW] (reason: [shards failed [[cluster-metricbeat-2018.06.17][0]] ...]). [2018-06-17T03:21:28,758][WARN][o.e.d.z.PublishClusterStateAction] [master] timed out waiting for all nodes to process published state [1996] (timeout [30s], pending nodes: [{data2}{99iDFQX-QMWiC9Og9NZB-Q}{s05h 0qgvRmuqmHJatHLY-g}{10.10.10.24}{10.10.10.24:9500}]) [2018-06-17T03:21:28,779][WARN][o.e.c.s.MasterService] [master] cluster state update task [cluster_reroute(async_shard_fetch)] took [30s] above the warn threshold of 30s [2018-06-17T03:21:28,828][INFO][o.e.c.r.a.AllocationService] [master] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[cluster-metricbeat-2018.06.17][0]] ...]). [2018-06-17T03:21:58,831][WARN][o.e.d.z.PublishClusterStateAction] [master] timed out waiting for all nodes to process published state [1997] (timeout [30s], pending nodes: [{data2}{99iDFQX-QMWiC9Og9NZB-Q}{s05h 0qgvRmuqmHJatHLY-g}{10.10.10.24}{10.10.10.24:9500}]) [2018-06-17T03:21:59,037][WARN][o.e.c.s.MasterService] [master] cluster state update task [shard-started shard id [[cluster-metricbeat-2018.06.17][0]], allocation id [PCnuW12oRt67drHdQVe50A], primary term [ 0], message [after peer recovery][shard id [[cluster-metricbeat-2018.06.17][0]], allocation id [PCnuW12oRt67drHdQVe50A], primary term [0], message [after peer recovery]]] took [30.2s] above the warn threshold of 30s [2018-06-17T03:22:43,438][INFO][o.e.c.r.a.AllocationService] [master] Cluster health status changed from [GREEN] to [YELLOW] (reason: [{data2}{99iDFQX-QMWiC9Og9NZB-Q}{s05h0qgvRmuqmHJatHLY-g}{10.10.10.24}{10.10. 10.24:9500} failed to ping, tried [3] times, each with maximum [30s] timeout]). [2018-06-17T03:22:45,306][INFO][o.e.c.s.MasterService] [master] zen-disco-node-failed({data2}{99iDFQX-QMWiC9Og9NZB-Q}{s05h0qgvRmuqmHJatHLY-g}{10.10.10.24}{10.10.10.24:9500}), reason(failed to ping, tried [3 ] times, each with maximum [30s] timeout)[{data2}{99iDFQX-QMWiC9Og9NZB-Q}{s05h0qgvRmuqmHJatHLY-g}{10.10.10.24}{10.10.10.24:9500} failed to ping, tried [3] times, each with maximum [30s] timeout], reason: removed {{data2}{99iDFQX-QMWiC9Og9NZB-Q}{s05h0qgvRmuqmHJatHLY-g}{10.10.10.24}{10.10.10.24:9500},} ``` ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [June 18, 2018, 2:45pm UTC](https://discuss.elastic.co/t/forcemerge-failed-java-lang-arrayindexoutofboundsexception/136317/4 "2018-06-18T14:45:18Z")
I think those logs might be enough... But if at some point you need to upload more, you can always share that on [gist.github.com](http://gist.github.com) for example and paste the link here.... @jpountz do you have an idea? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 18, 2018, 2:51pm UTC](https://discuss.elastic.co/t/forcemerge-failed-java-lang-arrayindexoutofboundsexception/136317/5 "2018-06-18T14:51:22Z")
It looks like you have problems with the cluster state not getting updated quickly enough across the cluster. Do you have very large mappings, mappings that are updated frequently or a lot of shards in the cluster? ---
**Author:** ![pfseth](https://avatars.discourse-cdn.com/v4/letter/p/e9bcb4/32.png) [@pfseth](https://discuss.elastic.co/u/pfseth)\ **Post date:** [June 18, 2018, 3:09pm UTC](https://discuss.elastic.co/t/forcemerge-failed-java-lang-arrayindexoutofboundsexception/136317/6 "2018-06-18T15:09:51Z")
> [@Christian\_Dahlqvist](#): > > mappings Do you have large mappings - No Mapping update frequently - No Lots of shards - Well, the total number of shards is 1608 ( 803 Primary & 805 Replica ) However, i forcemerge only the previous day's indices ( i.e total 6 indices = 36 shards ( 18 primary + 18 replica )) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 18, 2018, 3:14pm UTC](https://discuss.elastic.co/t/forcemerge-failed-java-lang-arrayindexoutofboundsexception/136317/7 "2018-06-18T15:14:14Z")
How many nodes? Are they under heavy load? ---
**Author:** ![pfseth](https://avatars.discourse-cdn.com/v4/letter/p/e9bcb4/32.png) [@pfseth](https://discuss.elastic.co/u/pfseth)\ **Post date:** [June 18, 2018, 3:19pm UTC](https://discuss.elastic.co/t/forcemerge-failed-java-lang-arrayindexoutofboundsexception/136317/8 "2018-06-18T15:19:48Z")
6 Nodes 2 - Coordinating nodes 4 - Master + Data ( any of the 4 nodes are master eligible with quorum set to 3 ) Well i run curator at 3:00 AM when the load is low ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [June 18, 2018, 3:54pm UTC](https://discuss.elastic.co/t/forcemerge-failed-java-lang-arrayindexoutofboundsexception/136317/9 "2018-06-18T15:54:04Z")
This exception should never happen, which suggests either a corruption or a JVM/Lucene bug. The corruption is a bit unlikely since we verify the integrity of each segment that participates in a merge before merging. Can you run CheckIndex on `[cluster-metricbeat-2018.06.17][0]` on this node? The command would look something like `java -cp libs/lucene-core-7.2.1.jar org.apache.lucene.index.CheckIndex data/nodes/0/indices/{uuid of cluster-metricbeat-2018.06.17}/0/index/`. Note that you need to either stop this node or close the `cluster-metricbeat-2018.06.17` index before running it. ---
**Author:** ![pfseth](https://avatars.discourse-cdn.com/v4/letter/p/e9bcb4/32.png) [@pfseth](https://discuss.elastic.co/u/pfseth)\ **Post date:** [June 18, 2018, 5:03pm UTC](https://discuss.elastic.co/t/forcemerge-failed-java-lang-arrayindexoutofboundsexception/136317/10 "2018-06-18T17:03:59Z")
Unfortunately, its in production so cannot stop the node. However i can close the index, but what is the expected output of this command ? > java -cp libs/lucene-core-7.2.1.jar org.apache.lucene.index.CheckIndex data/nodes/0/indices/{uuid of cluster-metricbeat-2018.06.17}/0/index/ ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [June 18, 2018, 5:23pm UTC](https://discuss.elastic.co/t/forcemerge-failed-java-lang-arrayindexoutofboundsexception/136317/11 "2018-06-18T17:23:20Z")
It will generate a report about the consistency of the Lucene index of this shard, giving us information about a potential corruption or Lucene bug. See [https://lucene.apache.org/core/7\_2\_1/core/org/apache/lucene/index/CheckIndex.html](https://lucene.apache.org/core/7_2_1/core/org/apache/lucene/index/CheckIndex.html). ---
**Author:** ![pfseth](https://avatars.discourse-cdn.com/v4/letter/p/e9bcb4/32.png) [@pfseth](https://discuss.elastic.co/u/pfseth)\ **Post date:** [June 18, 2018, 6:11pm UTC](https://discuss.elastic.co/t/forcemerge-failed-java-lang-arrayindexoutofboundsexception/136317/12 "2018-06-18T18:11:38Z")
Here is the output, CheckIndex of [cluster-metricbeat-2018.06.17](https://gist.github.com/smhulkund/f06aa28c5ec366afb8aa99b515693d82#file-cluster-metricbeat-2018-06-17-log) ---
**Author:** ![pfseth](https://avatars.discourse-cdn.com/v4/letter/p/e9bcb4/32.png) [@pfseth](https://discuss.elastic.co/u/pfseth)\ **Post date:** [June 21, 2018, 4:21pm UTC](https://discuss.elastic.co/t/forcemerge-failed-java-lang-arrayindexoutofboundsexception/136317/13 "2018-06-21T16:21:28Z")
@jpountz any luck with report ? is it some config issue or a bug ? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 19, 2018, 4:21pm UTC](https://discuss.elastic.co/t/forcemerge-failed-java-lang-arrayindexoutofboundsexception/136317/14 "2018-07-19T16:21:51Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.