Cannot access elasticsearch from my browser and curl command not working for public ip from server terminal

Elastic Stack Elasticsearch
1

Hi, Hope anyone helps me here...

I am having two ec2 instances and one of the instance having elasticsearch sucessfully running and accessible from browser.

However issue with second instance where i have elasticsearch installed and having same settings as previous one has and even i can get result when i run curl command on server using private ip address or localhost but when i do that for public ip, it thorws error - "Failed to connect at 9200: Connection timeout". network.host is set to 0.0.0.0 and bind address is also seems to be ok and no error in logs as elasticsearch is running on server but not accessible via public ip in curl command and also not accessible from browser as well.

Does anybody has any idea what could be reason for this? . I have tried every possible solutions but nothing worked. You have come across any such situation, please do let me know or you can ask me if you need any further details.

2

Please show the Elasticsearch logs from when you started the service.

3

You can see below screenshot. first one is the screenshot when i started first time and the second screenshot related to my last logs-

image
image.png1366×768 109 KB

image
image.png1366×768 89 KB

4

Please don't post pictures of text, they are difficult to read and some people may not be even able to see them.

5

ok here is the text-

[2019-02-03T19:43:00,209][INFO ][o.e.e.NodeEnvironment ] [JrSNOHN] using [1] data paths, mounts [[/ (/dev/nvme0n1p1)]], net usable_space [5.4gb], net total_space [9.6gb], types [ext4]
[2019-02-03T19:43:00,211][INFO ][o.e.e.NodeEnvironment ] [JrSNOHN] heap size [1007.3mb], compressed ordinary object pointers [true]
[2019-02-03T19:43:00,213][INFO ][o.e.n.Node ] [JrSNOHN] node name derived from node ID [JrSNOHNgQh2FawKwALtK8g]; set [node.name] to override
[2019-02-03T19:43:00,213][INFO ][o.e.n.Node ] [JrSNOHN] version[6.5.4], pid[5252], build[default/deb/d2ef93d/2018-12-17T21:17:40.758843Z], OS[Linux/4.4.0-1074-aws/amd64], JVM[Or$
[2019-02-03T19:43:00,213][INFO ][o.e.n.Node ] [JrSNOHN] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccup$
[2019-02-03T19:43:02,701][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [aggs-matrix-stats]
[2019-02-03T19:43:02,702][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [analysis-common]
[2019-02-03T19:43:02,702][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [ingest-common]
[2019-02-03T19:43:02,702][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [lang-expression]
[2019-02-03T19:43:02,702][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [lang-mustache]
[2019-02-03T19:43:02,702][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [lang-painless]
[2019-02-03T19:43:02,702][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [mapper-extras]
[2019-02-03T19:43:02,702][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [parent-join]
[2019-02-03T19:43:02,702][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [percolator]
[2019-02-03T19:43:02,702][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [rank-eval]
[2019-02-03T19:43:02,702][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [reindex]
[2019-02-03T19:43:02,702][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [repository-url]
[2019-02-03T19:43:02,702][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [transport-netty4]
[2019-02-03T19:43:02,702][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [tribe]
[2019-02-03T19:43:02,703][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-ccr]
[2019-02-03T19:43:02,703][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-core]
[2019-02-03T19:43:02,703][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-deprecation]
[2019-02-03T19:43:02,703][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-graph]
[2019-02-03T19:43:02,703][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-logstash]
[2019-02-03T19:43:02,703][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-ml]
[2019-02-03T19:43:02,703][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-monitoring]
[2019-02-03T19:43:02,703][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-rollup]
[2019-02-03T19:43:02,703][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-security]
[2019-02-03T19:43:02,703][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-sql]
[2019-02-03T19:43:02,703][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-upgrade]
[2019-02-03T19:43:02,703][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-watcher]
[2019-02-03T19:43:02,704][INFO ][o.e.p.PluginsService ] [JrSNOHN] no plugins loaded
[2019-02-03T19:43:07,244][INFO ][o.e.x.s.a.s.FileRolesStore] [JrSNOHN] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2019-02-03T19:43:07,903][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [JrSNOHN] [controller/5308] [Main.cc@109] controller (64 bit): Version 6.5.4 (Build b616085ef32393) Copyright (c) 2018 Ela$
[2019-02-03T19:43:08,655][DEBUG][o.e.a.ActionModule ] [JrSNOHN] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2019-02-03T19:43:09,021][INFO ][o.e.d.DiscoveryModule ] [JrSNOHN] using discovery type [zen] and host providers [settings]
[2019-02-03T19:43:10,390][INFO ][o.e.n.Node ] [JrSNOHN] initialized
[2019-02-03T19:43:10,390][INFO ][o.e.n.Node ] [JrSNOHN] starting ...
[2019-02-03T19:43:10,579][INFO ][o.e.t.TransportService ] [JrSNOHN] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2019-02-03T19:43:13,654][INFO ][o.e.c.s.MasterService ] [JrSNOHN] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {JrSNOHN}{JrSNOHNgQh2FawKwALtK8g}{NoJJhAXkRXimqfku6Ki9$
[2019-02-03T19:43:13,659][INFO ][o.e.c.s.ClusterApplierService] [JrSNOHN] new_master {JrSNOHN}{JrSNOHNgQh2FawKwALtK8g}{NoJJhAXkRXimqfku6Ki9SQ}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=205$
[2019-02-03T19:43:13,692][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [JrSNOHN] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2019-02-03T19:43:13,693][INFO ][o.e.n.Node ] [JrSNOHN] started
[2019-02-03T19:43:13,730][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [JrSNOHN] Failed to clear cache for realms []
[2019-02-03T19:43:13,806][INFO ][o.e.g.GatewayService ] [JrSNOHN] recovered [0] indices into cluster_state

6

Below one the latest logs-

[2019-02-03T20:57:49,245][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [percolator]
[2019-02-03T20:57:49,245][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [rank-eval]
[2019-02-03T20:57:49,245][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [reindex]
[2019-02-03T20:57:49,245][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [repository-url]
[2019-02-03T20:57:49,245][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [transport-netty4]
[2019-02-03T20:57:49,245][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [tribe]
[2019-02-03T20:57:49,245][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-ccr]
[2019-02-03T20:57:49,245][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-core]
[2019-02-03T20:57:49,245][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-deprecation]
[2019-02-03T20:57:49,245][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-graph]
[2019-02-03T20:57:49,246][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-logstash]
[2019-02-03T20:57:49,246][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-ml]
[2019-02-03T20:57:49,246][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-monitoring]
[2019-02-03T20:57:49,246][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-rollup]
[2019-02-03T20:57:49,246][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-security]
[2019-02-03T20:57:49,246][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-sql]
[2019-02-03T20:57:49,246][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-upgrade]
[2019-02-03T20:57:49,246][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-watcher]
[2019-02-03T20:57:49,246][INFO ][o.e.p.PluginsService ] [JrSNOHN] no plugins loaded
[2019-02-03T20:57:53,794][INFO ][o.e.x.s.a.s.FileRolesStore] [JrSNOHN] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2019-02-03T20:57:54,714][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [JrSNOHN] [controller/1538] [Main.cc@109] controller (64 bit): Version 6.5.4 (Build b616085ef32393) Copyright (c) 2018 Ela$
[2019-02-03T20:57:55,236][DEBUG][o.e.a.ActionModule ] [JrSNOHN] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2019-02-03T20:57:55,471][INFO ][o.e.d.DiscoveryModule ] [JrSNOHN] using discovery type [zen] and host providers [settings]
[2019-02-03T20:57:56,292][INFO ][o.e.n.Node ] [JrSNOHN] initialized
[2019-02-03T20:57:56,293][INFO ][o.e.n.Node ] [JrSNOHN] starting ...
[2019-02-03T20:57:56,470][INFO ][o.e.t.TransportService ] [JrSNOHN] publish_address {172.31.14.91:9300}, bound_addresses {[::]:9300}
[2019-02-03T20:57:56,490][INFO ][o.e.b.BootstrapChecks ] [JrSNOHN] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-02-03T20:57:59,593][INFO ][o.e.c.s.MasterService ] [JrSNOHN] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {JrSNOHN}{JrSNOHNgQh2FawKwALtK8g}{KRaGeqxcT4aPx2Y-Ec9g$
[2019-02-03T20:57:59,599][INFO ][o.e.c.s.ClusterApplierService] [JrSNOHN] new_master {JrSNOHN}{JrSNOHNgQh2FawKwALtK8g}{KRaGeqxcT4aPx2Y-Ec9gxg}{172.31.14.91}{172.31.14.91:9300}{ml.machine_memo$
[2019-02-03T20:57:59,634][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [JrSNOHN] publish_address {172.31.14.91:9200}, bound_addresses {[::]:9200}
[2019-02-03T20:57:59,635][INFO ][o.e.n.Node ] [JrSNOHN] started
[2019-02-03T20:57:59,946][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [JrSNOHN] Failed to clear cache for realms []
[2019-02-03T20:58:00,006][INFO ][o.e.l.LicenseService ] [JrSNOHN] license [c685a2d0-5bff-49d0-94fc-a5ae78da5162] mode [basic] - valid
[2019-02-03T20:58:00,023][INFO ][o.e.g.GatewayService ] [JrSNOHN] recovered [0] indices into cluster_state

7

When i curl using private ip i get correct output-

{
"name" : "JrSNOHN",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "bfdy4EQLQWWCJDkx3JT80w",
"version" : {
"number" : "6.5.4",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "d2ef93d",
"build_date" : "2018-12-17T21:17:40.758843Z",
"build_snapshot" : false,
"lucene_version" : "7.5.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}

But when i curl using public ip this what i get-

Failed to connect to 13.232.169.223 port 9200: Connection timed out

8

It's still binding to localhost, so perhaps your config changes are not correct?

Also, please format logs/config/output using the </> button, or markdown-style backticks. It makes it much easier to read.

9

No that's not the case if u see my last logs, you will see it is not bound to localhost- I had changed network.host to 0.0.0.0

JrSNOHN] publish_address {172.31.14.91:9200}, bound_addresses {[::]:9200}
[2019-02-03T20:57:59,635][INFO ][o.e.n.Node ] [JrSNOHN] started
[2019-02-03T20:57:59,946][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [JrSNOHN] Failed to clear cache for realms
[2019-02-03T20:58:00,006][INFO ][o.e.l.LicenseService ] [JrSNOHN] license [c685a2d0-5bff-49d0-94fc-a5ae78da5162] mode [basic] - valid
[2019-02-03T20:58:00,023][INFO ][o.e.g.GatewayService ] [

10

You have two different IPs?

11

Yes public and private IP. I can connect to private IP but not to public IP.

12

Right, cause Elasticsearch is not binding to the public IP. So what did you actually set in the config.

I hope you have access control on this cluster by the way. Exposing things to the internet is not a good idea.

13

Thanks for your concern. this is just for my test server..

But how is that possible possible because I have another server where it running perfectly fine having same configuration in .yml file of elasticsearh.

14

That's not really the point. It's bad practise.

It'd help if you posted our your config.
Please make sure you use the code format button, as mentioned above.

15

Just the default settings I have in .yml file except network.host points to 0.0.0.0. rest are untouched. I also added http cors setting there but that does not have any effect.

16 
#cluster.name: my-application
#node.name: node-1

#node.attr.rack: r1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
#bootstrap.memory_lock: true
network.host: 0.0.0.0
#http.port: 9200
#discovery.zen.ping.unicast.hosts: ["host1", "host2"]
#discovery.zen.minimum_master_nodes:
# Block initial recovery after a full cluster restart until N nodes are started:
#gateway.recover_after_nodes: 3
#action.destructive_requires_name: true
http.enabled: true
http.cors.enabled: true
http.cors.allow-origin: "*"
17

And what IPs are attached to the network interfaces on the host?

18

172.31.14.91- ens5
:127.0.0.1 -lo

i can see these two in ifconfig

19

Ok, then it cannot bind to that public IP if it's not attached to an interface on the host. So you will need to figure that part out.

20

but that is the same case with my another server. i can see same network interfaces with private ip of ens5 type and loopback of lo type there as well. no public ip attached there. but can connect to elasticsearch without any issue in that server