Tribe node had Java Internal Error

Elastic Stack Elasticsearch
1

We are setting up Tribe Node to do search against two Elasticsearch cluster in two different data centers. WE saw a java internal error in the tribe node logs.

The Elasticsearch version is 5.5.2.

The Java version is 1.8.0_91"

We are getting the following Java Internal Error

[2017-11-07T11:49:19,248][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [r00j5dn0c] fatal error in thread [elasticsearch[r00j5dn0c][search][T#9]], exiting
java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
at org.apache.lucene.codecs.blocktree.SegmentTermsEnumFrame.loadBlock(SegmentTermsEnumFrame.java:174) ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46]
at org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:507) ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46]
at org.apache.lucene.index.TermContext.build(TermContext.java:99) ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46]
at org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:198) ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46]
at org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:751) ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46]
at org.apache.lucene.search.BooleanWeight.(BooleanWeight.java:60) ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46]
at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:225) ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46]
at org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:751) ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46]
at org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:734) ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46]
at org.elasticsearch.search.internal.ContextIndexSearcher.createNormalizedWeight(ContextIndexSearcher.java:109) ~[elasticsearch-5.5.2.jar:5.5.2]
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:472) ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46]
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:388) ~[elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:108) ~[elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:248) ~[elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:263) ~[elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:330) ~[elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:327) ~[elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1544) ~[elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.5.2.jar:5.5.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_144]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_144]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]

The

The tribe nodes elasticsearch.yml

node.name: tribe_${HOSTNAME}
path.data: /opt/pki/elasticsearch_data
path.logs: /opt/pki/elasticsearch_logs
http.port: 9200
transport.tcp.port: 9300

tribe:
blocks:
write: true
metadata: true
c1:
cluster.name: cluster1
discovery.zen.ping.unicast.hosts: [ "r00j4un0c", "r00j4xn0c", "r00j4yn0c" ]
c2:
cluster.name: cluster2
discovery.zen.ping.unicast.hosts: [ "r00j8dn0c", "r00j8en0c", "r00j8jn0c" ]

2

This should not be related to the tribe node. This exception looks like a JVM fault due to a problem in mmap. It should appear on a data node but not on the tribe node since this node should not hold any shard. Are you using NFS or had issues with disk space recently ? Bottom line is that this is not an elasticsearch issue but rather an issue with the JVM or the memory subsystem. You can try to upgrade to the latest JVM and also checks the logs of your kernels to see if there are errors reported there.

3

We have plenty of space on the file system. I believe we are using the latest JVM or almost the latest.

4

And are you sure that you saw this error in the logs of the tribe node ? The tribe node does not perform any local search so this error should appear in a data node and the node should halt right after . This data node should be checked and not the tribe node which has nothing to do with this crash.

5

We had a data node running here before. We needed the server for the tribe node.

6

And did you clean up the directories that es used for the data node (/opt/pki/elasticsearch_data) before start ?
Can you try to start with a clean installation (no previous data) ?
The tribe node should not have any data locally.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.