Tribe node fails to launch due to missing hunspell

Elastic Stack Elasticsearch
1

Would really appricaite if someone could hint me as to why an attempt to launch a tribe node fails with ref. to non-existing hunspell.

How to I find out why the tribe thinks it needs such, when none of the nodes in either of my two clusters have hunspell nor do I delibrately ref hunspeel anywhere?

How to resolve this I wonder?
hunspell 2.1.1 plugin cant be found and installed...

$ service elasticsearch start
Starting elasticsearch: Exception in thread "main" java.security.AccessControlException: access denied ("java.io.FilePermission" "/usr/share/elasticsearch/config/hunspell" "read")
at java.security.AccessControlContext.checkPermission(Unknown Source)
at java.security.AccessController.checkPermission(Unknown Source)
at java.lang.SecurityManager.checkPermission(Unknown Source)
at java.lang.SecurityManager.checkRead(Unknown Source)
at sun.nio.fs.UnixPath.checkRead(Unknown Source)
at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(Unknown Source)
at sun.nio.fs.UnixFileSystemProvider.readAttributes(Unknown Source)
at sun.nio.fs.LinuxFileSystemProvider.readAttributes(Unknown Source)
at java.nio.file.Files.readAttributes(Unknown Source)
at java.nio.file.Files.isDirectory(Unknown Source)
at org.elasticsearch.indices.analysis.HunspellService.scanAndLoadDictionaries(HunspellService.java:127)
at org.elasticsearch.indices.analysis.HunspellService.(HunspellService.java:102)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at <<>>
at org.elasticsearch.node.Node.(Node.java:200)
at org.elasticsearch.tribe.TribeClientNode.(TribeClientNode.java:35)
at org.elasticsearch.tribe.TribeService.(TribeService.java:140)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at <<>>
at org.elasticsearch.node.Node.(Node.java:200)
at org.elasticsearch.node.Node.(Node.java:128)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:145)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:285)
<<>>
[FAILED]

2

Even installing hunspell dictionaries and dup such to ES config still fails the same way:

ls -la /usr/share/elasticsearch/config/hunspell
total 1076
drwxr-xr-x 2 elasticsearch elasticsearch 4096 Jan 30 11:01 .
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Jan 30 10:59 ..
-rw-r--r-- 1 elasticsearch elasticsearch 27449 Jan 30 11:01 en_GB.aff
-rw-r--r-- 1 elasticsearch elasticsearch 527349 Jan 30 11:01 en_GB.dic
-rw-r--r-- 1 elasticsearch elasticsearch 3114 Jan 30 11:01 en_US.aff
-rw-r--r-- 1 elasticsearch elasticsearch 532021 Jan 30 11:01 en_US.dic

why the F... wont the tribe node launch past this point 'beats' me :frowning:

3

Needless to say, it's NOT a permission issue for elasticsearch user on the mentioned path. Initially after installing ES the /usr/share/elasticsearch/config directory isn't found at all.

Question is why at all do a tribe node want this, when neither of my ES nodes has hunspell nor any mentioning nor specific references to any hunspell. Would it coming by eg. .kibana index or otherwise default logstash indicies somehow...

4

That looks weird indeed.

At least the error message seems to indicate a read error to the config dir.

I can confirm that you don't need to create any subdir in config dir. But config dir has to be readable by elasticsearch process. As you wrote it I guess you double checked the permission.

Can you describe all the steps you did and on which system you are running?

5

I'm on CentSO 6.7, added elasticsearch repo

[root@kibana ~]# cat /etc/yum.repos.d/elasticsearch.repo
[elasticsearch-2.0]
baseurl=http://packages.elastic.co/elasticsearch/2.x/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1

and did 'yum -y install elasticsearch' which installed ES 2.1.1, edited /etc/elasticsearch/elasticsearch.yml as a tribe node:

[root@kibana ~]# cat /etc/elasticsearch/elasticsearch.yml
node.name: kibana
network.host: 127.0.0.1

tribe:
perf:
cluster.name: perfcluster
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["n1", "n2", "n3",..., "nX"]
logs:
cluster.name: logcluster
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["ln1", "ln2",...,"lnX"]

, then trying to launch the service w/ service elasticsearch start, which failed w/above error. Then I added a config (cd /usr/share/elasticsearch/; mkdir config; chown elasticsearch:elasticsearch config) no change then tried adding a hunspell sub dir (mkdir config/hunspell; chown elasticsearch:elasticsearch config/hunspell) no change, did install hunspell dictionaries (yum -y install hunspell-en.noarch) and

cp /usr/share/myspell/en_GB.* config/hunspell/.
cp /usr/share/myspell/en_US.* config/hunspell/.
chown elasticsearch:elasticsearch config/hunspell/*

still no change, same error when launching the node:

[root@kibana ~]# cat /var/log/elasticsearch/elasticsearch.log
[2016-01-30 14:14:05,172][INFO ][node ] [kibana] version[2.1.1], pid[16261], build[40e2c53/2015-12-15T13:05:55
Z]
[2016-01-30 14:14:05,173][INFO ][node ] [kibana] initializing ...
[2016-01-30 14:14:05,226][INFO ][plugins ] [kibana] loaded , sites
[2016-01-30 14:14:06,651][INFO ][node ] [kibana/log] version[2.1.1], pid[16261], build[40e2c53/2015-12-15T13:0
5:55Z]
[2016-01-30 14:14:06,651][INFO ][node ] [kibana/log] initializing ...
[2016-01-30 14:14:06,705][INFO ][plugins ] [kibana/log] loaded , sites
[2016-01-30 14:14:07,630][ERROR][bootstrap ] Guice Exception: java.security.AccessControlException: access denied (
"java.io.FilePermission" "/usr/share/elasticsearch/config/hunspell" "read")
at java.security.AccessControlContext.checkPermission(Unknown Source)
at java.security.AccessController.checkPermission(Unknown Source)
...

I check the permission by 'su - elasticsearch -c "ls -la /usr/share/elasticsearch/config/"'
But why do ES as a tribe node expect to find nor check for hunspell at all, I wonder, since I haven't used hunspell on either of my ES cluster nodes?

6

Hm may have to read more on configuring tribes.... check this bug 14573 as well

7

Finally, this worked, adding an extra empty config per cluster:

$  cd /etc/elasticsearch/
$ touch tribe.yml
$ chown elasticsearch:elasticsearch tribe.yml

and then adding path.conf per tribe node:

 $ cat elasticsearch.yml 
 node.name: kibana
 network.host: 127.0.0.1
 tribe:
   perf:
    path.conf: /etc/elasticsearch/tribe.yml
    cluster.name: perfcluster
    discovery.zen.ping.multicast.enabled: false
    discovery.zen.ping.unicast.hosts: ["n1", "n2", ...,"nX"]
  log:
    path.conf: /etc/elasticsearch/tribe.yml
    cluster.name: logcluster
    discovery.zen.ping.multicast.enabled: false
    discovery.zen.ping.unicast.hosts: ["ln1",..."lnX"]

then service will launch :wink:

$ service elasticsearch start
Starting elasticsearch:                                    [  OK  ]

giving the indication of three nodes running on ports [9300-9302]:

$ netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 127.0.0.1:9200              0.0.0.0:*                   LISTEN      21479/java          
tcp        0      0 127.0.0.1:9300              0.0.0.0:*                   LISTEN      21479/java          
tcp        0      0 127.0.0.1:9301              0.0.0.0:*                   LISTEN      21479/java          
tcp        0      0 127.0.0.1:9302              0.0.0.0:*                   LISTEN      21479/java
1 Like
8

Minor adjustments to finally connect with my two clusters on each their subnets:

$ cat /etc/elasticsearch/elasticsearch.yml 
node.name: kibana
network.host: localhost
http.port: 9200
transport.tcp.port: 9300
path.data: /var/lib/elasticsearch/
path.logs: /var/log/elasticsearch/

tribe:
  # not necessarily but have my current .kibana index on perfcluster, all other indices are uniquely named across clusters :)
  # later I plan to give access for another kibana instance to logcluster only and have it's .kibana index there, so...
  on_conflict: prefer_perf
  perf:
    network.host: 10.45.70.62
    transport.tcp.port: 9300
    path.conf: /etc/elasticsearch/tribe.yml
    cluster.name: perfcluster
    discovery.zen.ping.multicast.enabled: false
    discovery.zen.ping.unicast.hosts: ["10.45.70.62", "n1", "n2",...,"nX"]
  log:
    path.conf: /etc/elasticsearch/tribe.yml
    network.host: 10.3.3.2
    transport.tcp.port: 9300
    cluster.name: logcluster
    discovery.zen.ping.multicast.enabled: false
    discovery.zen.ping.unicast.hosts: ["10.3.3.2", "ln1", "ln2",..., "lnX"]

Giving these nodes:

$ netstat -lntp
Active Internet connections (only servers)  
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 127.0.0.1:9200              0.0.0.0:*                   LISTEN      24301/java          
tcp        0      0 10.45.70.62:9300            0.0.0.0:*                   LISTEN      24301/java          
tcp        0      0 10.3.3.2:9300               0.0.0.0:*                   LISTEN      24301/java          
tcp        0      0 127.0.0.1:9300              0.0.0.0:*                   LISTEN      24301/java

Then I connect my kibana instance to localhost:9200 and see collectively all indicies across both clusters now :wink: