Cluster (connect 2 nodes)


(Salma Ait Lhaj) #1

Hi,

I want to connecte 2 nodes Elastic search to make a cluster, the two nodes are on a different servers. Can someone help me to do this ?


(Salma Ait Lhaj) #2

I can't find on "elasticsearch.yml" :

node.master:
node.data:
index.number_of_replicas:


(Salma Ait Lhaj) #3

I did :

Node 1 :
cluster.name: network-logs
node.name: network-1
network.host: 0.0.0.0
discovery.zen.ping.unicast.hosts: ["172.16.250.29"]

Node 2 :
cluster.name: network-logs
node.name: network-2
network.host: 0.0.0.0
discovery.zen.ping.unicast.hosts: ["172.16.250.29"]

How can I know that this works ?


(Jamesspi) #4

Hi @asalma,

Can you run on any of the two nodes you mention above?:

http://localhost:9200/_cat/nodes?pretty 

If both nodes show in the results, everything works as expected.

James


(Salma Ait Lhaj) #5

on the first node it worked but on the second I have :

curl: (7) Failed connect to localhost:9200; Connection refused


(Jamesspi) #6

Is elasticsearch running on the second node? Can you run:

netstat -pan | grep 9200

Also, did you check the logs in /var/log/elasticsearch/network-logs.log ?


(Salma Ait Lhaj) #7
[root@frghcslnetv12 data]# netstat -pan | grep 9200
[root@frghcslnetv12 data]# cd /var/log/elasticsearch
[root@frghcslnetv12 elasticsearch]# ls
elasticsearch_deprecation.log             network-logs_deprecation.log
elasticsearch_index_indexing_slowlog.log  network-logs_index_indexing_slowlog.log
elasticsearch_index_search_slowlog.log    network-logs_index_search_slowlog.log
elasticsearch.log                         network-logs.log
gc.log.0.current 

I started elasticsearch but i think it is not running


(Jamesspi) #8

Can you check what there is in network-logs.log please?


(Salma Ait Lhaj) #9

[2018-06-04T11:33:48,555][ERROR][o.e.b.Bootstrap          ] Exception
java.lang.IllegalStateException: Unable to access 'path.data' (/data/elasticsearch)
        at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:70) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:307) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:262) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Security.configure(Security.java:123) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:208) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) [elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) [elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) [elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.2.4.jar:6.2.4]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) [elasticsearch-6.2.4.jar:6.2.4]
Caused by: java.nio.file.AccessDeniedException: /data/elasticsearch
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:1.8.0_171]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_171]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_171]
        at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:1.8.0_171]
        at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_171]
        at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_171]
        at java.nio.file.Files.createDirectories(Files.java:767) ~[?:1.8.0_171]
        at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:421) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:68) ~[elasticsearch-6.2.4.jar:6.2.4]
        ... 12 more
[2018-06-04T11:33:48,566][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Unable to access 'path.data' (/data/elasticsearch)
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.2.4.jar:6.2.4]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) ~[elasticsearch-6.2.4.jar:6.2.4]
Caused by: java.lang.IllegalStateException: Unable to access 'path.data' (/data/elasticsearch)
        at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:70) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:307) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:262) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Security.configure(Security.java:123) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:208) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.4.jar:6.2.4]
        ... 6 more
Caused by: java.nio.file.AccessDeniedException: /data/elasticsearch
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:1.8.0_171]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_171]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_171]
        at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:1.8.0_171]
        at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_171]
        at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_171]
        at java.nio.file.Files.createDirectories(Files.java:767) ~[?:1.8.0_171]
        at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:421) ~[elasticsearch-6.2.4.jar:6.2.4]
        at 

... etc


(Jamesspi) #10

Right, you need to set the correct ownership on the elasticsearch data directory:

chown -R elasticsearch:elasticsearch /data/elasticsearch

Then start the elasticsearch service again.


(Salma Ait Lhaj) #11
[root@frghcslnetv12 data]# ls -l
total 20
drwxr-x--- 3 elasticsearch elasticsearch  4096 Jun  4 11:52 elasticsearch
drwx------ 2 root          root          16384 Jun  1 11:33 lost+found
[root@frghcslnetv12 elasticsearch]# curl http://localhost:9200/_cat/nodes?pretty
curl: (7) Failed connect to localhost:9200; Connection refused

(Jamesspi) #12

You need to give it a minute to start up - and confirm that it started correctly by checking the same log as before.


(Salma Ait Lhaj) #13
[2018-06-04T11:33:48,555][ERROR][o.e.b.Bootstrap          ] Exception
java.lang.IllegalStateException: Unable to access 'path.data' (/data/elasticsearch)
        at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:70) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:307) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:262) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Security.configure(Security.java:123) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:208) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) [elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) [elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) [elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.2.4.jar:6.2.4]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) [elasticsearch-6.2.4.jar:6.2.4]
Caused by: java.nio.file.AccessDeniedException: /data/elasticsearch
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:1.8.0_171]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_171]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_171]
        at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:1.8.0_171]
        at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_171]
        at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_171]
        at java.nio.file.Files.createDirectories(Files.java:767) ~[?:1.8.0_171]
        at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:421) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:68) ~[elasticsearch-6.2.4.jar:6.2.4]
        ... 12 more
[2018-06-04T11:33:48,566][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Unable to access 'path.data' (/data/elasticsearch)
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.2.4.jar:6.2.4]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) ~[elasticsearch-6.2.4.jar:6.2.4]
Caused by: java.lang.IllegalStateException: Unable to access 'path.data' (/data/elasticsearch)
        at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:70) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:307) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:262) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Security.configure(Security.java:123) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:208) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.4.jar:6.2.4]
        ... 6 more
Caused by: java.nio.file.AccessDeniedException: /data/elasticsearch
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:1.8.0_171]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_171]

I'm still having the same problem :confused:


(Jamesspi) #14

What are the permissions on /data?


(Salma Ait Lhaj) #15
drwxrwx---    4 log-admin log-admin 4096 Jun  4 11:31 data

(Jamesspi) #16

Ok, so you need to add the elasticsearch user as a member of the log-admin group:

usermod -aG log-admin elasticsearch

(Salma Ait Lhaj) #17

[root@frghcslnetv12 data]# curl http://localhost:9200/_cat/nodes?pretty
{
  "error" : {
    "root_cause" : [
      {
        "type" : "master_not_discovered_exception",
        "reason" : null
      }
    ],
    "type" : "master_not_discovered_exception",
    "reason" : null
  },
  "status" : 503
}

Node 2 doesn't work
(Jamesspi) #18

logs?


(Salma Ait Lhaj) #19
[2018-06-04T11:33:48,555][ERROR][o.e.b.Bootstrap          ] Exception
java.lang.IllegalStateException: Unable to access 'path.data' (/data/elasticsearch)
        at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:70) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:307) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:262) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Security.configure(Security.java:123) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:208) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) [elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) [elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) [elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.2.4.jar:6.2.4]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) [elasticsearch-6.2.4.jar:6.2.4]
Caused by: java.nio.file.AccessDeniedException: /data/elasticsearch
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:1.8.0_171]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_171]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_171]
        at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:1.8.0_171]
        at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_171]
        at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_171]
        at java.nio.file.Files.createDirectories(Files.java:767) ~[?:1.8.0_171]
        at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:421) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:68) ~[elasticsearch-6.2.4.jar:6.2.4]
        ... 12 more
[2018-06-04T11:33:48,566][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Unable to access 'path.data' (/data/elasticsearch)
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.2.4.jar:6.2.4]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) ~[elasticsearch-6.2.4.jar:6.2.4]
Caused by: java.lang.IllegalStateException: Unable to access 'path.data' (/data/elasticsearch)
        at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:70) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:307) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:262) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Security.configure(Security.java:123) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:208) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.4.jar:6.2.4]
        ... 6 more
Caused by: java.nio.file.AccessDeniedException: /data/elasticsearch
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:1.8.0_171]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_171]

(Jamesspi) #20

I think you are posting the same log every time (the timestamp is the same). Are you going to the end of the file when checking?