[Help!] Tribe node startup fails if tribe.on_conflict option specified

Elastic Stack Elasticsearch
1

Hello.

I am now using elasticsearch 5.3.2 with x-pack 5.3.2 , however, when tribe.on_conflict is specified , the startup of elasticsearch fails as below.

[2017-04-29T17:55:43,497][ERROR][o.e.b.Bootstrap          ] Exception
org.elasticsearch.ElasticsearchException: Failed to load plugin class [org.elasticsearch.xpack.XPackPlugin]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:394) ~[elasticsearch-5.3.2.jar:5.3.2]
        at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:346) ~[elasticsearch-5.3.2.jar:5.3.2]
        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:129) ~[elasticsearch-5.3.2.jar:5.3.2]
        at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-5.3.2.jar:5.3.2]
        at org.elasticsearch.node.Node.newTribeClientNode(Node.java:995) ~[elasticsearch-5.3.2.jar:5.3.2]
        at org.elasticsearch.node.Node.lambda$new$6(Node.java:399) ~[elasticsearch-5.3.2.jar:5.3.2]
        at org.elasticsearch.tribe.TribeService.<init>(TribeService.java:227) ~[elasticsearch-5.3.2.jar:5.3.2]
        at org.elasticsearch.node.Node.<init>(Node.java:398) ~[elasticsearch-5.3.2.jar:5.3.2]
        at org.elasticsearch.node.Node.<init>(Node.java:242) ~[elasticsearch-5.3.2.jar:5.3.2]
        at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:242) ~[elasticsearch-5.3.2.jar:5.3.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[elasticsearch-5.3.2.jar:5.3.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) [elasticsearch-5.3.2.jar:5.3.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) [elasticsearch-5.3.2.jar:5.3.2]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) [elasticsearch-5.3.2.jar:5.3.2]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:58) [elasticsearch-5.3.2.jar:5.3.2]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.3.2.jar:5.3.2]
        at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.3.2.jar:5.3.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.3.2.jar:5.3.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.3.2.jar:5.3.2]
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_131]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:383) ~[elasticsearch-5.3.2.jar:5.3.2]
        ... 18 more
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/usr/share/elasticsearch/config/x-pack/system_key" "read")
        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_131]
        at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_131]
        at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_131]
        at java.lang.SecurityManager.checkRead(SecurityManager.java:888) ~[?:1.8.0_131]
        at sun.nio.fs.UnixPath.checkRead(UnixPath.java:795) ~[?:?]
        at sun.nio.fs.UnixFileSystemProvider.checkAccess(UnixFileSystemProvider.java:290) ~[?:?]
        at java.nio.file.Files.exists(Files.java:2385) ~[?:1.8.0_131]
        at org.elasticsearch.xpack.security.crypto.CryptoService.readSystemKey(CryptoService.java:159) ~[?:?]
        at org.elasticsearch.xpack.security.crypto.CryptoService.<init>(CryptoService.java:112) ~[?:?]
        at org.elasticsearch.xpack.security.Security.<init>(Security.java:209) ~[?:?]
        at org.elasticsearch.xpack.XPackPlugin.<init>(XPackPlugin.java:200) ~[?:?]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_131]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:383) ~[elasticsearch-5.3.2.jar:5.3.2]
        ... 18 more

I have configured tribe node as below.

node.master: false
node.data: false
node.ingest: false

tribe.on_conflict: prefer_tdc1
tribe:
  tdc1:
    cluster.name: dc1
    node.name: ${HOSTNAME}
    network.host: 172.16.x.xx
    transport.host: 172.16.x.xx
    discovery.zen.ping.unicast.hosts: ["172.16.x.xx", "172.16.x.xx", "172.16.x.xx"]
  tdc2:
    cluster.name: dc2
    node.name: ${HOSTNAME}
    network.host: 10.xxx.xx.x
    transport.host: 10.xxx.xx.x
    discovery.zen.ping.unicast.hosts: ["10.xxx.xx.x", "10.xxx.xx.x", "10.xxx.xx.x"]

I have installed x-pack from file and there was no error during the installation.
Is there any thing that I have missed with configuration?

2

Looking at the log , below directory path doesn't even exist when x-pack is installed.

/usr/share/elasticsearch/config/x-pack

Is this something that you have to create?

3

By following the document, error still exists event though I create the system_key.

[sysadmin@xxxx ~]$ sudo ls -ltr /usr/share/elasticsearch/config/x-pack
合計 4
-rw-------. 1 elasticsearch elasticsearch 128 4月 29 21:50 system_key

4

This sort of error (FilePermission on a config file) is generally caused by having mismatch between your path.conf setting, and where various files have been put.
The Elasticsearch security manager restricts the process to only being able to read files that are in the specified locations (config dir, data dir, etc), so if your process tries to read files outside of those locations, it will fail.

Because tribe is involved, it could be more complicated than that, but the starting point is to look at things like:

  • Does your tribe node have a path.conf set?
  • What directory is your elasticsearch.yml stored in?
  • How does that compare to the path in the error message?
1 Like
5

@TimV

Thank you for the reply. Your first suggestion did the fix. I had to explicitly specify path.conf in elasticsearch.yml for tribe node. (Not in the tribe node stanza)

# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs

path.conf: /etc/elasticsearch

Did this behavior changed in 5.3.2?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.