# Fatal error occuring on all nodes within the hour **URL:** \ **Category:** Elasticsearch\ **Created:** [May 4, 2020, 7:30am UTC](https://discuss.elastic.co/t/fatal-error-occuring-on-all-nodes-within-the-hour/230933 "2020-05-04T07:30:47Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![faxm0dem](https://avatars.discourse-cdn.com/v4/letter/f/c77e96/32.png) [@faxm0dem](https://discuss.elastic.co/u/faxm0dem)\ **Post date:** [May 4, 2020, 7:30am UTC](https://discuss.elastic.co/t/fatal-error-occuring-on-all-nodes-within-the-hour/230933/1 "2020-05-04T07:30:47Z")
Using 6.8.6 from the oss rpm for redhat and jvm openjdk-1.8.0.252.b09-2.el7\_8 on Centos 7, we've been getting crashes every second day now on almost all nodes within the hour. I managed to catch the log during the crash: ```auto [2020-05-04T08:31:08,354][INFO][o.e.c.m.MetaDataMappingService] [node84-indexer] [syslog-2020.05.04/-QEZ7dwzSD-2TYIaGUI2cA] update_mapping [_doc] [2020-05-04T08:31:09,045][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [ccsvli84-coloss-indexer] fatal error in thread [elasticsearch[ccsvli84-coloss-indexer][search][T#15]], exiting java.lang.InternalError: java.io.IOException: Stream closed at sun.util.locale.provider.BreakIteratorProviderImpl.getBreakInstance(BreakIteratorProviderImpl.java:178) ~[?:?] at sun.util.locale.provider.BreakIteratorProviderImpl.getSentenceInstance(BreakIteratorProviderImpl.java:148) ~[?:?] at java.text.BreakIterator.createBreakInstance(BreakIterator.java:574) ~[?:1.8.0_242] at java.text.BreakIterator.createBreakInstance(BreakIterator.java:553) ~[?:1.8.0_242] at java.text.BreakIterator.getBreakInstance(BreakIterator.java:544) ~[?:1.8.0_242] at java.text.BreakIterator.getSentenceInstance(BreakIterator.java:531) ~[?:1.8.0_242] at org.apache.lucene.search.uhighlight.BoundedBreakIteratorScanner.getSentence(BoundedBreakIteratorScanner.java:151) ~[elasticsearch-6.8.6.jar:7.7.2 d4c30fc2856154f2c1fefc589eb7cd070a415b94 - janhoy - 2019-05-28 23:31:35] at org.elasticsearch.search.fetch.subphase.highlight.UnifiedHighlighter.getBreakIterator(UnifiedHighlighter.java:193) ~[elasticsearch-6.8.6.jar:6.8.6] at org.elasticsearch.search.fetch.subphase.highlight.UnifiedHighlighter.highlight(UnifiedHighlighter.java:118) ~[elasticsearch-6.8.6.jar:6.8.6] at org.elasticsearch.search.fetch.subphase.highlight.HighlightPhase.hitExecute(HighlightPhase.java:107) ~[elasticsearch-6.8.6.jar:6.8.6] at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:167) ~[elasticsearch-6.8.6.jar:6.8.6] at org.elasticsearch.search.SearchService.lambda$executeFetchPhase$3(SearchService.java:541) ~[elasticsearch-6.8.6.jar:6.8.6] at org.elasticsearch.search.SearchService$3.doRun(SearchService.java:381) ~[elasticsearch-6.8.6.jar:6.8.6] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.8.6.jar:6.8.6] at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) ~[elasticsearch-6.8.6.jar:6.8.6] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[elasticsearch-6.8.6.jar:6.8.6] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.8.6.jar:6.8.6] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_242] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_242] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242] Caused by: java.io.IOException: Stream closed at java.io.BufferedInputStream.getInIfOpen(BufferedInputStream.java:159) ~[?:1.8.0_242] at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[?:1.8.0_242] at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[?:1.8.0_242] at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[?:1.8.0_242] at java.io.FilterInputStream.read(FilterInputStream.java:107) ~[?:1.8.0_242] at sun.util.locale.provider.RuleBasedBreakIterator.readFile(RuleBasedBreakIterator.java:462) ~[?:?] at sun.util.locale.provider.RuleBasedBreakIterator.readTables(RuleBasedBreakIterator.java:375) ~[?:?] at sun.util.locale.provider.RuleBasedBreakIterator.(RuleBasedBreakIterator.java:321) ~[?:?] at sun.util.locale.provider.BreakIteratorProviderImpl.getBreakInstance(BreakIteratorProviderImpl.java:169) ~[?:?] ... 19 more ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 4, 2020, 7:54am UTC](https://discuss.elastic.co/t/fatal-error-occuring-on-all-nodes-within-the-hour/230933/2 "2020-05-04T07:54:50Z")
It seems to be a IO exception. What type of storage are you using? What type of hardware is this cluster deployed on? ---
**Author:** ![faxm0dem](https://avatars.discourse-cdn.com/v4/letter/f/c77e96/32.png) [@faxm0dem](https://discuss.elastic.co/u/faxm0dem)\ **Post date:** [May 4, 2020, 8:07am UTC](https://discuss.elastic.co/t/fatal-error-occuring-on-all-nodes-within-the-hour/230933/3 "2020-05-04T08:07:38Z")
Bare-metal servers with local hardware RAID storage. I checked system and hardware logs and there is nothing suspicious. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 4, 2020, 3:14pm UTC](https://discuss.elastic.co/t/fatal-error-occuring-on-all-nodes-within-the-hour/230933/4 "2020-05-04T15:14:59Z")
I suspect a JDK bug here: JDK8 converts a nonfatal `IOException` into a fatal `InternalError` [here](https://github.com/openjdk/jdk/blob/jdk8-b120/jdk/src/share/classes/sun/util/locale/provider/BreakIteratorProviderImpl.java#L177-L179): ```nohighlight } catch (IOException | MissingResourceException | IllegalArgumentException e) { throw new InternalError(e.toString(), e); } ``` in JDK10 [it doesn't](https://github.com/openjdk/jdk/blob/jdk-10+0/jdk/src/java.base/share/classes/sun/util/locale/provider/BreakIteratorProviderImpl.java#L182-L184): ```nohighlight } catch (MissingResourceException | IllegalArgumentException e) { throw new InternalError(e.toString(), e); } ``` I haven't dug in depth into the reasons surrounding this change. (edit: added links to the JDK source) ---
**Author:** ![faxm0dem](https://avatars.discourse-cdn.com/v4/letter/f/c77e96/32.png) [@faxm0dem](https://discuss.elastic.co/u/faxm0dem)\ **Post date:** [May 4, 2020, 3:17pm UTC](https://discuss.elastic.co/t/fatal-error-occuring-on-all-nodes-within-the-hour/230933/5 "2020-05-04T15:17:29Z")
wow, very interesting find! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 1, 2020, 3:17pm UTC](https://discuss.elastic.co/t/fatal-error-occuring-on-all-nodes-within-the-hour/230933/6 "2020-06-01T15:17:30Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.