Failed to connect to ip-myip port 80: Connection refused

Elastic Stack Elasticsearch
1

I have installed elasticsearch 7 on Ubuntu 18. I have 3 nodes, all master eligible.

This is my configuration:

First server: 172.31.1.1

# /etc/elasticsearch/elasticsearch.yml
cluster.name: es-cluster
node.name: es-1

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

network.host: 172.31.1.1
http.port: 9200

discovery.seed_hosts: ["172.31.1.1", "172.31.1.2", "172.31.1.3"]
cluster.initial_master_nodes:["es-1", "es-2", "es-3"]

gateway.recover_after_nodes: 2
gateway.expected_nodes: 3

action.destructive_requires_name: true

Second server: 172.31.1.2

node.name: es-2
network.host: 172.31.1.2
# everything else same as first server

Third server: 172.31.1.3

node.name: es-3
network.host: 172.31.1.3
# everything else same as first server

This is /etc/default/elasticsearch

# /etc/default/elasticsearch
JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64
ES_PATH_CONF=/etc/elasticsearch
RESTART_ON_UPGRADE=true
ES_STARTUP_SLEEP_TIME=5
MAX_OPEN_FILES=65535

START_DAEMON=true
ES_HEAP_SIZE=2g
# I am not sure if I should add these or not?
# LOG_DIR=/var/log/elasticsearch
# DATA_DIR=/var/lib/elasticsearch
# WORK_DIR=/tmp/elasticsearch
# CONF_DIR=/etc/elasticsearch
# CONF_FILE=/etc/elasticsearch/elasticsearch.yml

# I have tried adding these variables but got error saying these are no longer in use
# ES_USER=elasticsearch
# ES_GROUP=elasticsearch

And the file: /usr/lib/systemd/system/elasticsearch.service

# /usr/lib/systemd/system/elasticsearch.service

# some more config not shown... I have not changed this file... all default values
[Service]
RuntimeDirectory=elasticsearch
PrivateTmp=true
Environment=ES_HOME=/usr/share/elasticsearch
Environment=ES_PATH_CONF=/etc/elasticsearch
Environment=PID_DIR=/var/run/elasticsearch
EnvironmentFile=-/etc/default/elasticsearch
WorkingDirectory=/usr/share/elasticsearch

User=elasticsearch
Group=elasticsearch

ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid

I have changed the owner of the following files to elasticsearch:elasticsearch

sudo chown elasticsearch:elasticsearch -R /usr/share/elasticsearch
sudo chown elasticsearch:elasticsearch -R /var/log/elasticsearch
sudo chown elasticsearch:elasticsearch -R /var/lib/elasticsearch
sudo chown elasticsearch:elasticsearch -R /etc/default/elasticsearch
sudo chown elasticsearch:elasticsearch -R /etc/elasticsearch

Now I start the first node (only the first node)

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service

First issue is that nothing is written to log file

If I check the status of elasticsearch I have:

Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2019-05-26 11:12:25 AEST; 46min ago
     Docs: http://www.elastic.co
 Main PID: 5014 (java)
    Tasks: 40 (limit: 4915)
   CGroup: /system.slice/elasticsearch.service
           ├─5014 /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java ...
           └─5088 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

[INFO ][o.e.d.DiscoveryModule    ] [ip-172-31-1-1] using discovery type [zen] and seed h
[INFO ][o.e.n.Node               ] [ip-172-31-1-1] initialized
[INFO ][o.e.n.Node               ] [ip-172-31-1-1] starting ...
[INFO ][o.e.t.TransportService   ] [ip-172-31-1-1] publish_address {172.31.1.1:9300},
[INFO ][o.e.b.BootstrapChecks    ] [ip-172-31-1-1] bound or publishing to a non-loopback
[INFO ][o.e.c.c.Coordinator      ] [ip-172-31-1-1] cluster UUID [...]
[INFO ][o.e.c.s.MasterService    ] [ip-172-31-1-1] elected-as-master ([1] nodes joined)[
[INFO ][o.e.c.s.ClusterApplierService] [ip-172-31-1-1] master node changed {previous [],
[INFO ][o.e.h.AbstractHttpServerTransport] [ip-172-31-1-1] publish_address {172.31.1.1
[INFO ][o.e.n.Node               ] [ip-172-31-1-1] started

I have no idea why it is saying:

publish_address {172.31.1.1:9300} // shouldn't be 9200?

If I check the health of cluster

curl http://ip-172-31-1-1:9200/_cat/health
somenumber 02:05:56 es-cluster red 1 1 0 0 0 0 0 0 - NaN%

If I try:

curl http://ip-172-31-1-1/:9200/_cat/nodes
curl: (7) Failed to connect to ip-172-31-1-1 port 80: Connection refused

If I try:

journalctl -f

[INFO ][o.e.c.s.ClusterApplierService] [ip-172-31-11-1] master node changed {previous [], current [{ip-172-31-1-1}{...}{...}{172.31.1.1}{172.31.1.1:9300}{ml.machine_memory=..., xpack.installed=true, ml.max_open_jobs=20}]}, term: 27, version: 45, reason: Publication{term=27, version=45}
[INFO ][o.e.h.AbstractHttpServerTransport] [ip-172-31-1-1] publish_address {172.31.1.1:9200}, bound_addresses {172.31.1.1:9200}
[INFO ][o.e.n.Node               ] [ip-172-31-1-1] started
2

Take out the / before the port#

3

Did you install using the DEB?

4

@rugnel, thanks for your comment. I updated the question... I have commented out port 9200 and I realized I was reading the wring log file... I have included my log exception.

5

@warkolm: thanks a lot. I have updated the question. I removed port : 9200 from
elasticsearch.yml, because I believe it should have been 9300? So the port is now comment out.

I have also included the log file (I was reading the wrong log file)

This is how I installed elasticseach:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo add-apt-repository "deb https://artifacts.elastic.co/packages/7.x/apt stable main"
sudo apt-get update
sudo apt-get install elasticsearch

6

Which port?

So why did you do the chown?

7

in /etc/elasticsearch/elasticsearch.yml

I had:

http.port: 9200

Now I removed it... I think that was one problem...

As I was not able to read the log file (I was reading the wrong log files) I was not sure what I am doing... I followed this question: https://stackoverflow.com/questions/40766301/run-elastic-search-as-root-user/50617708#50617708

8

I made 2 modifications to: /etc/elasticsearch/elasticsearch.yml

network.host: 172.31.1.1
# http.port: 9200 <-- commented

# only included other servers in discovery.seed_hosts 
discovery.seed_hosts: ["172.31.1.2", "172.31.1.3"]

Then I kept first node running. stopped elasticsearch on second and third nodes, deleted their data path (content of /var/lib/elasticsearch in my case) and restarted elasticsearch on second and third node and they are now joining the cluster.