Can't connect to Elasticsearch at <Publicip>:9200 - Logs don't show any errors

Elastic Stack Elasticsearch
1

This worked, thanks. I am running into a different issue. Elasticsearch status says it's active, but I can't connect at http://:9200. My elasticsearch logs doesn't show any issue.

This is the yaml file which was working for me on a different instance:

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
node.data : true
network.host : 0.0.0.0
discovery.seed_hosts : []
cluster.initial_master_nodes : [10.50.11.77]

And this is the tailed logs:

tail -f /var/log/elasticsearch/elasticsearch.log

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]

at java.lang.Thread.run(Thread.java:833) [?:?]

[2022-10-05T23:50:13,021][INFO ][o.e.i.g.DatabaseNodeService] [ip-10-50-11-77] retrieve geoip database [GeoLite2-Country.mmdb] from [.geoip_databases] to [/tmp/geoip-databases/vr34RjMDQHGmLx60I5HrCw/GeoLite2-Country.mmdb.tmp.gz]

[2022-10-05T23:50:13,031][INFO ][o.e.i.g.DatabaseNodeService] [ip-10-50-11-77] retrieve geoip database [GeoLite2-ASN.mmdb] from [.geoip_databases] to [/tmp/geoip-databases/vr34RjMDQHGmLx60I5HrCw/GeoLite2-ASN.mmdb.tmp.gz]

[2022-10-05T23:50:13,038][INFO ][o.e.i.g.DatabaseNodeService] [ip-10-50-11-77] retrieve geoip database [GeoLite2-City.mmdb] from [.geoip_databases] to [/tmp/geoip-databases/vr34RjMDQHGmLx60I5HrCw/GeoLite2-City.mmdb.tmp.gz]

[2022-10-05T23:50:13,234][INFO ][o.e.c.r.a.AllocationService] [ip-10-50-11-77] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.ds-.logs-deprecation.elasticsearch-default-2022.10.05-000001][0]]]).

[2022-10-05T23:50:13,771][INFO ][o.e.i.g.DatabaseNodeService] [ip-10-50-11-77] successfully reloaded changed geoip database file [/tmp/geoip-databases/vr34RjMDQHGmLx60I5HrCw/GeoLite2-Country.mmdb]

[2022-10-05T23:50:13,906][INFO ][o.e.i.g.DatabaseNodeService] [ip-10-50-11-77] successfully reloaded changed geoip database file [/tmp/geoip-databases/vr34RjMDQHGmLx60I5HrCw/GeoLite2-ASN.mmdb]

[2022-10-05T23:50:14,877][INFO ][o.e.i.g.DatabaseNodeService] [ip-10-50-11-77] successfully reloaded changed geoip database file [/tmp/geoip-databases/vr34RjMDQHGmLx60I5HrCw/GeoLite2-City.mmdb]
2

Can you share your Elasticsearch log from startup please?

3

Do you mean all the logs from var/log/elasticsearch/elasticsearch.log?

4

Which version are you using? node.data has been removed in Elasticsearch 8.x If you are using Elasticsearch 8.x. It's been deprecated since 7.9. Use node.roles instead.

And yes, please share your Elasticsearch log, because there was an exception stacktrace based on your log snippet.

5

I am using 7.17

6 
[2022-10-06T00:03:29,946][INFO ][o.e.p.PluginsService     ] [ip-10-50-11-77] no plugins loaded
[2022-10-06T00:03:30,035][INFO ][o.e.e.NodeEnvironment    ] [ip-10-50-11-77] using [1] data paths, mounts [[/ (/dev/root)]], net usable_space [90.1gb], net total_space [96.7gb], types [ext4]
[2022-10-06T00:03:30,036][INFO ][o.e.e.NodeEnvironment    ] [ip-10-50-11-77] heap size [4gb], compressed ordinary object pointers [true]
[2022-10-06T00:03:30,217][INFO ][o.e.n.Node               ] [ip-10-50-11-77] node name [ip-10-50-11-77], node ID [vr34RjMDQHGmLx60I5HrCw], cluster name [elasticsearch], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data>
[2022-10-06T00:03:40,323][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [ip-10-50-11-77] [controller/79368] [Main.cc@122] controller (64 bit): Version 7.17.6 (Build ea0305ae2a3b09) Copyright (c) 2022 Elasticsearch BV
[2022-10-06T00:03:41,764][INFO ][o.e.x.s.a.s.FileRolesStore] [ip-10-50-11-77] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2022-10-06T00:03:43,305][INFO ][o.e.i.g.ConfigDatabases  ] [ip-10-50-11-77] initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/etc/elasticsearch/ingest-geoip] for changes
[2022-10-06T00:03:43,306][INFO ][o.e.i.g.DatabaseNodeService] [ip-10-50-11-77] initialized database registry, using geoip-databases directory [/tmp/geoip-databases/vr34RjMDQHGmLx60I5HrCw]
[2022-10-06T00:03:44,538][INFO ][o.e.t.NettyAllocator     ] [ip-10-50-11-77] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=>
[2022-10-06T00:03:44,599][INFO ][o.e.i.r.RecoverySettings ] [ip-10-50-11-77] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2022-10-06T00:03:44,685][INFO ][o.e.d.DiscoveryModule    ] [ip-10-50-11-77] using discovery type [zen] and seed hosts providers [settings]
[2022-10-06T00:03:45,759][INFO ][o.e.g.DanglingIndicesState] [ip-10-50-11-77] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-10-06T00:03:47,175][INFO ][o.e.n.Node               ] [ip-10-50-11-77] initialized
[2022-10-06T00:03:47,176][INFO ][o.e.n.Node               ] [ip-10-50-11-77] starting ...
[2022-10-06T00:03:47,191][INFO ][o.e.x.s.c.f.PersistentCache] [ip-10-50-11-77] persistent cache index loaded
[2022-10-06T00:03:47,192][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [ip-10-50-11-77] deprecation component started
[2022-10-06T00:03:47,306][INFO ][o.e.t.TransportService   ] [ip-10-50-11-77] publish_address {10.50.11.77:9300}, bound_addresses {0.0.0.0:9300}
[2022-10-06T00:03:48,200][INFO ][o.e.b.BootstrapChecks    ] [ip-10-50-11-77] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-10-06T00:03:48,299][INFO ][o.e.c.c.Coordinator      ] [ip-10-50-11-77] cluster UUID [S6SmAd6RS0SIElyFlQi0WA]
[2022-10-06T00:03:48,480][INFO ][o.e.c.s.MasterService    ] [ip-10-50-11-77] elected-as-master ([1] nodes joined)[{ip-10-50-11-77}{vr34RjMDQHGmLx60I5HrCw}{3sRm9pnPStCO4Kb07_7OoA}{10.50.11.77}{10.50.11.77:9300}{cdfhilmrstw} elect leader, _BECOME_MASTER_TASK_, _FINISH_EL>
[2022-10-06T00:03:48,616][INFO ][o.e.c.s.ClusterApplierService] [ip-10-50-11-77] master node changed {previous [], current [{ip-10-50-11-77}{vr34RjMDQHGmLx60I5HrCw}{3sRm9pnPStCO4Kb07_7OoA}{10.50.11.77}{10.50.11.77:9300}{cdfhilmrstw}]}, term: 15, version: 241, reason: P>
[2022-10-06T00:03:48,720][INFO ][o.e.h.AbstractHttpServerTransport] [ip-10-50-11-77] publish_address {10.50.11.77:9200}, bound_addresses {0.0.0.0:9200}
[2022-10-06T00:03:48,720][INFO ][o.e.n.Node               ] [ip-10-50-11-77] started
[2022-10-06T00:03:49,468][WARN ][r.suppressed             ] [ip-10-50-11-77] path: /.kibana_task_manager/_search, params: {ignore_unavailable=true, index=.kibana_task_manager, track_total_hits=true}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:713) [elasticsearch-7.17.6.jar:7.17.6]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:400) [elasticsearch-7.17.6.jar:7.17.6]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:745) [elasticsearch-7.17.6.jar:7.17.6]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:497) [elasticsearch-7.17.6.jar:7.17.6]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.performPhaseOnShard(AbstractSearchAsyncAction.java:308) [elasticsearch-7.17.6.jar:7.17.6]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:244) [elasticsearch-7.17.6.jar:7.17.6]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:454) [elasticsearch-7.17.6.jar:7.17.6]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:199) [elasticsearch-7.17.6.jar:7.17.6]
        at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1048) [elasticsearch-7.17.6.jar:7.17.6]
        at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:763) [elasticsearch-7.17.6.jar:7.17.6]
        at org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$6(TransportSearchAction.java:399) [elasticsearch-7.17.6.jar:7.17.6]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.6.jar:7.17.6]
        at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:112) [elasticsearch-7.17.6.jar:7.17.6]
        at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77) [elasticsearch-7.17.6.jar:7.17.6]
        at org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:487) [elasticsearch-7.17.6.jar:7.17.6]
        at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:285) [elasticsearch-7.17.6.jar:7.17.6]
        at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:101) [elasticsearch-7.17.6.jar:7.17.6]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:186) [elasticsearch-7.17.6.jar:7.17.6]
        at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) [elasticsearch-7.17.6.jar:7.17.6]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:184) [elasticsearch-7.17.6.jar:7.17.6]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:145) [x-pack-security-7.17.6.jar:7.17.6]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:184) [elasticsearch-7.17.6.jar:7.17.6]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:161) [elasticsearch-7.17.6.jar:7.17.6]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82) [elasticsearch-7.17.6.jar:7.17.6]

Caused by: org.elasticsearch.action.NoShardAvailableActionException
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:532) ~[elasticsearch-7.17.6.jar:7.17.6]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:479) [elasticsearch-7.17.6.jar:7.17.6]
        ... 84 more
[2022-10-06T00:45:30,665][ERROR][o.e.i.g.GeoIpDownloader  ] [ip-10-50-11-77] exception during geoip databases update
org.elasticsearch.ElasticsearchException: not all primary shards of [.geoip_databases] index are active
        at org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:137) ~[ingest-geoip-7.17.6.jar:7.17.6]
        at org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:284) [ingest-geoip-7.17.6.jar:7.17.6]
        at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:100) [ingest-geoip-7.17.6.jar:7.17.6]
        at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:46) [ingest-geoip-7.17.6.jar:7.17.6]
        at org.elasticsearch.persistent.NodePersistentTasksExecutor$1.doRun(NodePersistentTasksExecutor.java:42) [elasticsearch-7.17.6.jar:7.17.6]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) [elasticsearch-7.17.6.jar:7.17.6]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.17.6.jar:7.17.6]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-10-06T00:45:31,185][INFO ][o.e.i.g.DatabaseNodeService] [ip-10-50-11-77] retrieve geoip database [GeoLite2-Country.mmdb] from [.geoip_databases] to [/tmp/geoip-databases/vr34RjMDQHGmLx60I5HrCw/GeoLite2-Country.mmdb.tmp.gz]
[2022-10-06T00:45:31,191][INFO ][o.e.i.g.DatabaseNodeService] [ip-10-50-11-77] retrieve geoip database [GeoLite2-ASN.mmdb] from [.geoip_databases] to [/tmp/geoip-databases/vr34RjMDQHGmLx60I5HrCw/GeoLite2-ASN.mmdb.tmp.gz]
[2022-10-06T00:45:31,194][INFO ][o.e.i.g.DatabaseNodeService] [ip-10-50-11-77] retrieve geoip database [GeoLite2-City.mmdb] from [.geoip_databases] to [/tmp/geoip-databases/vr34RjMDQHGmLx60I5HrCw/GeoLite2-City.mmdb.tmp.gz]
[2022-10-06T00:45:31,606][INFO ][o.e.c.r.a.AllocationService] [ip-10-50-11-77] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.ds-.logs-deprecation.elasticsearch-default-2022.10.05-000001][0]]]).
[2022-10-06T00:45:31,957][INFO ][o.e.i.g.DatabaseNodeService] [ip-10-50-11-77] successfully reloaded changed geoip database file [/tmp/geoip-databases/vr34RjMDQHGmLx60I5HrCw/GeoLite2-Country.mmdb]
[2022-10-06T00:45:32,136][INFO ][o.e.i.g.DatabaseNodeService] [ip-10-50-11-77] successfully reloaded changed geoip database file [/tmp/geoip-databases/vr34RjMDQHGmLx60I5HrCw/GeoLite2-ASN.mmdb]
[2022-10-06T00:45:34,217][INFO ][o.e.i.g.DatabaseNodeService] [ip-10-50-11-77] successfully reloaded changed geoip database file [/tmp/geoip-databases/vr34RjMDQHGmLx60I5HrCw/GeoLite2-City.mmdb]
7

Do I need to open port 9300?

8

Same problem here, try the solution.

9

That didn't work for me. The site is still not reachable. It just cleared the error from the log.

10

Do you have other logs following the logs you have shared?

Based on the logs you have shared so far, I don't see any issue that might stop Elasticsearch from starting up. Have you tried to connect to Elasticsearch directly, e.g. using curl?

You can ignore the geoip related errors. They shouldn't have any impact in getting Elasticsearch started.

11

These are all the logs. Curl with curl -X GET http://localhost:9200/ or with the private ip of the instance works fine. But with the public IP, connection is refused. Port 9200 is open, so I had thought it would work. Did I miss anything in the yml config I shared?

12

I re-added port 5601 and 9200 to my AWS instance security group, but this time I set the source to 0.0.0.0/0 and it works.

13

Can you provide more context?

Your cluster is running in a cloud provider, but which one? You didn't say.

Since you talked bout security group, I'm assuming that you are running it in AWS.

If you want to allow external access to your instance you need to configure your security group and network acl to allow access only from the desired ip address.

For example, if you want to access your cluster using it's public IP from your home, you would need to configure the security group and network acl to use allow access from your home public ip address.

Your issue is related to how your cloud provider limit access from external IPs, so you should look in the documentation of your cloud provider.

In elasticsearch.yml you just need to configure it to listen on 0.0.0.0, everything else is configured in your cloud provider.

One more thing, do not expose your cluster to the public internet without security enabled, if you do, it is just a matter of time that someone else will have access to it.

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.