ELK curl: (7) Failed to connect to localhost port 9200: Connection refused

Hi There,

I met some problem while installing elk on 3 different server on Google cloud Platform.
First, I build 2 servers based on Debian GNU/Linux 9 (stretch),
after installing elasticsearch and kibana respectively,
I can $ curl -X GET http://localhost:9200 and get the result.
{
"name" : "elastic",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "fV_t9HRxTl2vrvZLF3EESA",
"version" : {
"number" : "7.1.1",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "7a013de",
"build_date" : "2019-05-23T14:04:00.380842Z",
"build_snapshot" : false,
"lucene_version" : "8.0.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}

Then because I want tocurl -X GET http://35.225.4.142:9200 through the internet and get the result like above situation.
So, the config I only modified at elasticsearch.yml as below:
network.host: 0.0.0.0
http.port: 9200

But when run sudo systemctl restart elasticsearch
sudo systemctl status elasticsearch

Then get error as below:
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2019-06-06 06:18:04 UTC; 978ms ago
Docs: http://www.elastic.co
Process: 13477 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, stat
us=78)
Main PID: 13477 (code=exited, status=78)

Jun 06 06:17:48 elastic systemd[1]: Started Elasticsearch.
Jun 06 06:17:48 elastic elasticsearch[13477]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0
and will likely be removed in a future release.
Jun 06 06:18:04 elastic systemd[1]: elasticsearch.service: Main process exited, code=exited, status=78/n/a
Jun 06 06:18:04 elastic systemd[1]: elasticsearch.service: Unit entered failed state.
Jun 06 06:18:04 elastic systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

Did anybody encounter the same problem and fixed?
Please let me know
Thank you

Elasticsearch failed to start after some changes you made in the configuration. Can you share the output of journalctl -u elasticsearch.service with us ? This will contain necessary information as to what might went wrong. ( Please use </> or (```) to format the log output , you can use the preview panel on the right to see how your post looks like

I guess you should change:
network.host: 0.0.0.0
to
network.host: 127.0.0.1

This is what i get:

yclin1822@elastic:~$ journalctl -u elasticsearch.service
-- Logs begin at Thu 2019-06-06 06:37:07 UTC, end at Thu 2019-06-06 06:41:10 UTC. --
Jun 06 06:39:08 elastic systemd[1]: Started Elasticsearch.
Jun 06 06:39:08 elastic elasticsearch[1650]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 a
nd will likely be removed in a future release.
Jun 06 06:40:51 elastic systemd[1]: Stopping Elasticsearch...
Jun 06 06:40:52 elastic systemd[1]: Stopped Elasticsearch.
Jun 06 06:40:52 elastic systemd[1]: Started Elasticsearch.
Jun 06 06:40:52 elastic elasticsearch[1840]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 a
nd will likely be removed in a future release.
Jun 06 06:41:08 elastic systemd[1]: elasticsearch.service: Main process exited, code=exited, status=78/n/a
Jun 06 06:41:08 elastic systemd[1]: elasticsearch.service: Unit entered failed state.
Jun 06 06:41:08 elastic systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.

sorry, i revise the log already.

Can you share elasticsearch logs?

This is all the elasticsearch log folder

sudo ls -l /var/log/elasticsearch/
total 788
-rw-r--r-- 1 elasticsearch elasticsearch      0 Jun  6 06:39 elasticsearch_audit.json
-rw-r--r-- 1 elasticsearch elasticsearch      0 Jun  6 06:39 elasticsearch_deprecation.json
-rw-r--r-- 1 elasticsearch elasticsearch      0 Jun  6 06:39 elasticsearch_deprecation.log
-rw-r--r-- 1 elasticsearch elasticsearch      0 Jun  6 06:39 elasticsearch_index_indexing_slowlog.json
-rw-r--r-- 1 elasticsearch elasticsearch      0 Jun  6 06:39 elasticsearch_index_indexing_slowlog.log
-rw-r--r-- 1 elasticsearch elasticsearch      0 Jun  6 06:39 elasticsearch_index_search_slowlog.json
-rw-r--r-- 1 elasticsearch elasticsearch      0 Jun  6 06:39 elasticsearch_index_search_slowlog.log
-rw-r--r-- 1 elasticsearch elasticsearch  24207 Jun  6 06:43 elasticsearch.log
-rw-r--r-- 1 elasticsearch elasticsearch  46505 Jun  6 06:43 elasticsearch_server.json
-rw-r--r-- 1 elasticsearch elasticsearch 270304 Jun  6 07:05 gc.log
-rw-r--r-- 1 elasticsearch elasticsearch 287918 Jun  6 06:40 gc.log.00
-rw-r--r-- 1 elasticsearch elasticsearch 168936 Jun  6 06:41 gc.log.01

indent preformatted text by 4 spaces

I tail the first 10 line from gc.log

yclin1822@elastic:~$ sudo tail -10 /var/log/elasticsearch/gc.log
[2019-06-06T07:07:59.369+0000][1997][safepoint ] Leaving safepoint region
[2019-06-06T07:07:59.369+0000][1997][safepoint ] Total time for which application threads were stopped: 0.0001376 seconds, Stopping threads took: 0.0000380 seconds
[2019-06-06T07:08:16.371+0000][1997][safepoint ] Application time: 17.0026062 seconds
[2019-06-06T07:08:16.371+0000][1997][safepoint ] Entering safepoint region: Cleanup
[2019-06-06T07:08:16.372+0000][1997][safepoint ] Leaving safepoint region
[2019-06-06T07:08:16.372+0000][1997][safepoint ] Total time for which application threads were stopped: 0.0001488 seconds, Stopping threads took: 0.0000447 seconds
[2019-06-06T07:09:07.379+0000][1997][safepoint ] Application time: 51.0073445 seconds
[2019-06-06T07:09:07.379+0000][1997][safepoint ] Entering safepoint region: Cleanup
[2019-06-06T07:09:07.379+0000][1997][safepoint ] Leaving safepoint region
[2019-06-06T07:09:07.379+0000][1997][safepoint ] Total time for which application threads were stopped: 0.0001421 seconds, Stopping threads took: 0.0000459 seconds

How about the elasticsearch.log contents ? ( not just 10 lines though, but a bit more and covering an attempt to start elasticsearch with the configuration that fails.)

It won't show error when i restart the service,
but curl Still shows connection refused.

OK, here it is.

[2019-06-06T07:21:40,282][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-security]
[2019-06-06T07:21:40,283][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-sql]
[2019-06-06T07:21:40,289][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-watcher]
[2019-06-06T07:21:40,290][INFO ][o.e.p.PluginsService     ] [elastic] no plugins loaded
[2019-06-06T07:21:46,960][INFO ][o.e.x.s.a.s.FileRolesStore] [elastic] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2019-06-06T07:21:48,205][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [elastic] [controller/2768] [Main.cc@109] controller (64 bit): Version 7.1.1 (Build fd619a36eb77df) Copyright (c) 2019 Elasticsearch BV
[2019-06-06T07:21:49,219][DEBUG][o.e.a.ActionModule       ] [elastic] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2019-06-06T07:21:49,663][INFO ][o.e.d.DiscoveryModule    ] [elastic] using discovery type [zen] and seed hosts providers [settings]
[2019-06-06T07:21:50,836][INFO ][o.e.n.Node               ] [elastic] initialized
[2019-06-06T07:21:50,837][INFO ][o.e.n.Node               ] [elastic] starting ...
[2019-06-06T07:21:51,009][INFO ][o.e.t.TransportService   ] [elastic] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2019-06-06T07:21:51,072][WARN ][o.e.b.BootstrapChecks    ] [elastic] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2019-06-06T07:21:51,079][INFO ][o.e.c.c.Coordinator      ] [elastic] cluster UUID [E-50p4vLRJCVoZj7YUH1pA]
[2019-06-06T07:21:51,090][INFO ][o.e.c.c.ClusterBootstrapService] [elastic] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2019-06-06T07:21:51,291][INFO ][o.e.c.s.MasterService    ] [elastic] elected-as-master ([1] nodes joined)[{elastic}{YPSBX2hLRTOlmdIcCK5nRg}{H_Z68oxWTrKyouorRFjzfw}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=7847481344, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 3, version: 19, reason: master node changed {previous [], current [{elastic}{YPSBX2hLRTOlmdIcCK5nRg}{H_Z68oxWTrKyouorRFjzfw}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=7847481344, xpack.installed=true, ml.max_open_jobs=20}]}
[2019-06-06T07:21:51,470][INFO ][o.e.c.s.ClusterApplierService] [elastic] master node changed {previous [], current [{elastic}{YPSBX2hLRTOlmdIcCK5nRg}{H_Z68oxWTrKyouorRFjzfw}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=7847481344, xpack.installed=true, ml.max_open_jobs=20}]}, term: 3, version: 19, reason: Publication{term=3, version=19}
[2019-06-06T07:21:51,531][INFO ][o.e.h.AbstractHttpServerTransport] [elastic] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}
[2019-06-06T07:21:51,532][INFO ][o.e.n.Node               ] [elastic] started
[2019-06-06T07:21:51,742][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [elastic] Failed to clear cache for realms [[]]
[2019-06-06T07:21:51,815][INFO ][o.e.l.LicenseService     ] [elastic] license [5ed0de7d-a8d8-478a-8886-e9add0975f35] mode [basic] - valid
[2019-06-06T07:21:51,834][INFO ][o.e.g.GatewayService     ] [elastic] recovered [0] indices into cluster_state

This log output is with you using network.host: 127.0.0.1 though. We know that elasticsearch starts successfully like this and the reason that curl gives you a connection refused when trying to connect is elasticsearch binds to the localhost (127.0.0.1) and by definition is not accessible from outside the server.

What we need is the logs from when you set network.host: 0.0.0.0 and then elasticsearch fails to start.

1 Like
[2019-06-06T07:32:49,963][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [rank-eval]
[2019-06-06T07:32:49,964][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [reindex]
[2019-06-06T07:32:49,965][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [repository-url]
[2019-06-06T07:32:49,965][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [transport-netty4]
[2019-06-06T07:32:49,966][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-ccr]
[2019-06-06T07:32:49,966][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-core]
[2019-06-06T07:32:49,967][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-deprecation]
[2019-06-06T07:32:49,967][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-graph]
[2019-06-06T07:32:49,967][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-ilm]
[2019-06-06T07:32:49,968][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-logstash]
[2019-06-06T07:32:49,968][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-ml]
[2019-06-06T07:32:49,968][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-monitoring]
[2019-06-06T07:32:49,969][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-rollup]
[2019-06-06T07:32:49,970][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-security]
[2019-06-06T07:32:49,971][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-sql]
[2019-06-06T07:32:49,971][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-watcher]
[2019-06-06T07:32:49,975][INFO ][o.e.p.PluginsService     ] [elastic] no plugins loaded
[2019-06-06T07:32:55,743][INFO ][o.e.x.s.a.s.FileRolesStore] [elastic] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2019-06-06T07:32:57,310][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [elastic] [controller/3296] [Main.cc@109] controller (64 bit): Version 7.1.1 (Build fd619a36eb77df) Copyright (c) 2019 Elasticsearch BV
[2019-06-06T07:32:58,352][DEBUG][o.e.a.ActionModule       ] [elastic] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2019-06-06T07:32:58,780][INFO ][o.e.d.DiscoveryModule    ] [elastic] using discovery type [zen] and seed hosts providers [settings]
[2019-06-06T07:32:59,869][INFO ][o.e.n.Node               ] [elastic] initialized
[2019-06-06T07:32:59,870][INFO ][o.e.n.Node               ] [elastic] starting ...
[2019-06-06T07:33:00,039][INFO ][o.e.t.TransportService   ] [elastic] publish_address {10.128.0.3:9300}, bound_addresses {[::]:9300}
[2019-06-06T07:33:00,050][INFO ][o.e.b.BootstrapChecks    ] [elastic] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-06-06T07:33:00,086][ERROR][o.e.b.Bootstrap          ] [elastic] node validation exception
[1] bootstrap checks failed
[1]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2019-06-06T07:33:00,093][INFO ][o.e.n.Node               ] [elastic] stopping ...
[2019-06-06T07:33:00,122][INFO ][o.e.n.Node               ] [elastic] stopped
[2019-06-06T07:33:00,122][INFO ][o.e.n.Node               ] [elastic] closing ...
[2019-06-06T07:33:00,146][INFO ][o.e.n.Node               ] [elastic] closed
[2019-06-06T07:33:00,149][INFO ][o.e.x.m.p.NativeController] [elastic] Native controller process has stopped - no new native processes can be started

see: https://www.elastic.co/guide/en/elasticsearch/reference/7.1/_discovery_configuration_check.html

1 Like

This time really works. Thank you so much.:smiley:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.