Configuring Elasticsearch 7 cluster

Elastic Stack Elasticsearch
1

I have three server and I have installed elasticsearch on all of them.

In the elasticsearch.yml I have the following config:

first server: 172.31.1.1

# /stc/elasticsearch.elasticsearch.yml
cluster.name: es-cluster
node.name: data-1
network.host: 172.31.1.1
http.port: 9200
network.host:
discovery.seed_hosts: ["172.31.1.1", "172.31.1.2", "172.31.1.3"]
cluster.initial_master_nodes:["data-1", "data-2", "data-3"]

gateway.recover_after_nodes: 2
gateway.expected_nodes: 3
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
# some more default values

second server: 172.31.1.2

node.name: data-2
network.host: 172.31.1.2
# everything else same as first server

second server: 172.31.1.3

node.name: data-3
network.host: 172.31.1.3
# everything else same as first server

Now when I run: curl http://data-1:9200/_cluster/stats

I notice that the cluster size is 1. if I run the same curl command on a different server, again the cluster size is 1 and the cluster_name is the same, but cluster_uuid is different...

It seems that nodes are not able to connect to each other...

I am using Ubuntu server, for node name I edited /etc/hostname and changed the host name to
data-1 and used the same name in elasticsearch.yml... but it seems that cluster nodes are not able to communicate with one another...

Update 1

curl http://data-1:9200/_cat/health
prod-es-cluster red 1 1 0 0 0 0 0 0 - NaN%

curl http://data-1/:9200/_cat/nodes
172.31.1.1 8 38 1 0.03 0.03 0.00 mdi * ip-172-31-1-1

I just check the log and it's strange...

sudo cat /var/log/elasticsearch/elasticsearch.log
[INFO ][o.e.n.Node               ] [data-1] stopping ...
[INFO ][o.e.x.w.WatcherService   ] [data-1] stopping watch service, reason [shutdown initiated]
[INFO ][o.e.x.m.p.l.CppLogMessageHandler] [data-1] [controller/19159] [Main.cc@148] Ml controller exiting
[INFO ][o.e.x.m.p.NativeController] [data-1] Native controller process has stopped - no new native processes can be started
[INFO ][o.e.n.Node               ] [data-1] stopped
[INFO ][o.e.n.Node               ] [data-1] closing ...
[INFO ][o.e.n.Node               ] [data-1] closed

That is all written to the log file... after restarting Elasticsearch nothing new gets added to the log file... it feels like I am either reading the wrong log file or I have put the configuration in the wrong .yml file

Update 2

This is how I start the service:

sudo /bin/systemctl enable elasticsearch.service

Update 3

As pointed out by @hamidbayat, I think there is permission issues. Only super user have permission to /var/lib/elasticsearch

If I try to run elastic search as my ubuntu user I will get this error:

ubuntu@my-ip:~$ /usr/share/elasticsearch/bin/elasticsearch
/usr/share/elasticsearch/bin/elasticsearch-env: line 73: 
/etc/default/elasticsearch: Permission denied

If I try to run it as root, I will get this error:

ubuntu@myip:~$ sudo /usr/share/elasticsearch/bin/elasticsearch
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[data][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [data-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.1.0.jar:7.1.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.1.0.jar:7.1.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.1.0.jar:7.1.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.1.0.jar:7.1.0]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.1.0.jar:7.1.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.1.0.jar:7.1.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.1.0.jar:7.1.0]
Caused by: java.lang.RuntimeException: can not run elasticsearch as root
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:102) ~[elasticsearch-7.1.0.jar:7.1.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:169) ~[elasticsearch-7.1.0.jar:7.1.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.1.0.jar:7.1.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.1.0.jar:7.1.0]
        ... 6 more
2

Is it a typo? This blank line:

network.host:

What are the logs?

3

This means that the first time these nodes started they formed separate clusters. Perhaps the config was different the first time around. You cannot merge two separate clusters together.

Are these nodes empty of data? If so, I think the simplest path forwards is to leave data-1 alone, but to shut down data-2 and data-3, wipe their data paths, and restart them.

4

@dadoonet: thanks for your message, yes that was a typo, fixed. It looks like a permission issue... I have not fixed the problem yet but I think the scope of the problem is narrowed down now.

5

@DavidTurner: thanks a lot. I think I have some permission issue as elasticsearch is not writing to log file... I have updated the question... I will follow your suggestion once I get the permission issue sorted.

6

@dadoonet, I have updated the question... it seems like a permission issue as elasticsearch is not able to write to log files/

7

@DavidTurner: I have updated the question, seem like a permission issue to write to log file.

8

You have changed your question to a completely different question which isn't very helpful. Anyone coming to this thread later will be mystified by how the answers you've already received don't match up any more, which is a shame because they might have helped another reader if only you'd left the question intact. Please would you set it back to how it was originally, and ask your new question separately?

2 Likes
9

@DavidTurner: apologies, I was only thinking about solving my own problem... did not think about it... I have reverted the question.

10

@DavidTurner: how to delete the data path? at the moment my data path (/var/lib/elasticsearch) contains nodes directory... should I delete nodes directory and it's content?

11

Yes, that'd do it.

1 Like
12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.