# Tribe node had Java Internal Error **URL:** \ **Category:** Elasticsearch\ **Created:** [November 7, 2017, 6:17pm UTC](https://discuss.elastic.co/t/tribe-node-had-java-internal-error/106734 "2017-11-07T18:17:56Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![ajstark123](https://avatars.discourse-cdn.com/v4/letter/a/73ab20/32.png) [@ajstark123](https://discuss.elastic.co/u/ajstark123)\ **Post date:** [November 7, 2017, 6:17pm UTC](https://discuss.elastic.co/t/tribe-node-had-java-internal-error/106734/1 "2017-11-07T18:17:56Z")
We are setting up Tribe Node to do search against two Elasticsearch cluster in two different data centers. WE saw a java internal error in the tribe node logs. The Elasticsearch version is 5.5.2. The Java version is 1.8.0\_91" We are getting the following Java Internal Error > [2017-11-07T11:49:19,248][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [r00j5dn0c] fatal error in thread [elasticsearch[r00j5dn0c][search][T#9]], exiting > java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code > at org.apache.lucene.codecs.blocktree.SegmentTermsEnumFrame.loadBlock(SegmentTermsEnumFrame.java:174) ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46] > at org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:507) ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46] > at org.apache.lucene.index.TermContext.build(TermContext.java:99) ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46] > at org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:198) ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46] > at org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:751) ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46] > at org.apache.lucene.search.BooleanWeight.(BooleanWeight.java:60) ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46] > at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:225) ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46] > at org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:751) ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46] > at org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:734) ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46] > at org.elasticsearch.search.internal.ContextIndexSearcher.createNormalizedWeight(ContextIndexSearcher.java:109) ~[elasticsearch-5.5.2.jar:5.5.2] > at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:472) ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46] > at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:388) ~[elasticsearch-5.5.2.jar:5.5.2] > at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:108) ~[elasticsearch-5.5.2.jar:5.5.2] > at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:248) ~[elasticsearch-5.5.2.jar:5.5.2] > at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:263) ~[elasticsearch-5.5.2.jar:5.5.2] > at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:330) ~[elasticsearch-5.5.2.jar:5.5.2] > at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:327) ~[elasticsearch-5.5.2.jar:5.5.2] > at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.5.2.jar:5.5.2] > at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1544) ~[elasticsearch-5.5.2.jar:5.5.2] > at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[elasticsearch-5.5.2.jar:5.5.2] > at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.5.2.jar:5.5.2] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0\_144] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0\_144] > at java.lang.Thread.run(Thread.java:748) [?:1.8.0\_144] The The tribe nodes elasticsearch.yml > node.name: tribe\_${HOSTNAME} > path.data: /opt/pki/elasticsearch\_data > path.logs: /opt/pki/elasticsearch\_logs > http.port: 9200 > transport.tcp.port: 9300 > > tribe: > blocks: > write: true > metadata: true > c1: > cluster.name: cluster1 > discovery.zen.ping.unicast.hosts: ["r00j4un0c", "r00j4xn0c", "r00j4yn0c"] > c2: > cluster.name: cluster2 > discovery.zen.ping.unicast.hosts: ["r00j8dn0c", "r00j8en0c", "r00j8jn0c"] ---
**Author:** ![jimczi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jimczi/32/47985_2.png) [@jimczi](https://discuss.elastic.co/u/jimczi)\ **Post date:** [November 8, 2017, 11:46am UTC](https://discuss.elastic.co/t/tribe-node-had-java-internal-error/106734/2 "2017-11-08T11:46:59Z")
This should not be related to the tribe node. This exception looks like a JVM fault due to a problem in mmap. It should appear on a data node but not on the tribe node since this node should not hold any shard. Are you using NFS or had issues with disk space recently ? Bottom line is that this is not an elasticsearch issue but rather an issue with the JVM or the memory subsystem. You can try to upgrade to the latest JVM and also checks the logs of your kernels to see if there are errors reported there. ---
**Author:** ![ajstark123](https://avatars.discourse-cdn.com/v4/letter/a/73ab20/32.png) [@ajstark123](https://discuss.elastic.co/u/ajstark123)\ **Post date:** [November 8, 2017, 9:45pm UTC](https://discuss.elastic.co/t/tribe-node-had-java-internal-error/106734/3 "2017-11-08T21:45:12Z")
We have plenty of space on the file system. I believe we are using the latest JVM or almost the latest. ---
**Author:** ![jimczi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jimczi/32/47985_2.png) [@jimczi](https://discuss.elastic.co/u/jimczi)\ **Post date:** [November 9, 2017, 11:19pm UTC](https://discuss.elastic.co/t/tribe-node-had-java-internal-error/106734/4 "2017-11-09T23:19:03Z")
And are you sure that you saw this error in the logs of the tribe node ? The tribe node does not perform any local search so this error should appear in a data node and the node should halt right after . This data node should be checked and not the tribe node which has nothing to do with this crash. ---
**Author:** ![ajstark123](https://avatars.discourse-cdn.com/v4/letter/a/73ab20/32.png) [@ajstark123](https://discuss.elastic.co/u/ajstark123)\ **Post date:** [November 10, 2017, 1:27pm UTC](https://discuss.elastic.co/t/tribe-node-had-java-internal-error/106734/5 "2017-11-10T13:27:26Z")
We had a data node running here before. We needed the server for the tribe node. ---
**Author:** ![jimczi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jimczi/32/47985_2.png) [@jimczi](https://discuss.elastic.co/u/jimczi)\ **Post date:** [November 10, 2017, 2:07pm UTC](https://discuss.elastic.co/t/tribe-node-had-java-internal-error/106734/6 "2017-11-10T14:07:23Z")
And did you clean up the directories that es used for the data node (/opt/pki/elasticsearch\_data) before start ? Can you try to start with a clean installation (no previous data) ? The tribe node should not have any data locally. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 8, 2017, 2:08pm UTC](https://discuss.elastic.co/t/tribe-node-had-java-internal-error/106734/7 "2017-12-08T14:08:10Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.