Cannot access elasticsearch from my browser and curl command not working for public ip from server terminal

Arvind_Singh · February 3, 2019, 9:32pm

Hi, Hope anyone helps me here...

I am having two ec2 instances and one of the instance having elasticsearch sucessfully running and accessible from browser.

However issue with second instance where i have elasticsearch installed and having same settings as previous one has and even i can get result when i run curl command on server using private ip address or localhost but when i do that for public ip, it thorws error - "Failed to connect at 9200: Connection timeout". network.host is set to 0.0.0.0 and bind address is also seems to be ok and no error in logs as elasticsearch is running on server but not accessible via public ip in curl command and also not accessible from browser as well.

Does anybody has any idea what could be reason for this? . I have tried every possible solutions but nothing worked. You have come across any such situation, please do let me know or you can ask me if you need any further details.

warkolm · February 3, 2019, 9:43pm

Please show the Elasticsearch logs from when you started the service.

Arvind_Singh · February 3, 2019, 9:48pm

You can see below screenshot. first one is the screenshot when i started first time and the second screenshot related to my last logs-

warkolm · February 3, 2019, 9:48pm

Please don't post pictures of text, they are difficult to read and some people may not be even able to see them.

Arvind_Singh · February 3, 2019, 9:51pm

ok here is the text-

[2019-02-03T19:43:00,209][INFO ][o.e.e.NodeEnvironment [2019-02-03T19:43:00,211][INFO ][o.e.e.NodeEnvironment [2019-02-03T19:43:00,213][INFO ][o.e.n.Node [2019-02-03T19:43:00,213][INFO ][o.e.n.Node [2019-02-03T19:43:00,213][INFO ][o.e.n.Node [2019-02-03T19:43:02,701][INFO ][o.e.p.PluginsService [2019-02-03T19:43:02,702][INFO ][o.e.p.PluginsService [2019-02-03T19:43:02,702][INFO ][o.e.p.PluginsService [2019-02-03T19:43:02,702][INFO ][o.e.p.PluginsService [2019-02-03T19:43:02,702][INFO ][o.e.p.PluginsService [2019-02-03T19:43:02,702][INFO ][o.e.p.PluginsService [2019-02-03T19:43:02,702][INFO ][o.e.p.PluginsService [2019-02-03T19:43:02,702][INFO ][o.e.p.PluginsService [2019-02-03T19:43:02,702][INFO ][o.e.p.PluginsService [2019-02-03T19:43:02,702][INFO ][o.e.p.PluginsService [2019-02-03T19:43:02,702][INFO ][o.e.p.PluginsService [2019-02-03T19:43:02,702][INFO ][o.e.p.PluginsService [2019-02-03T19:43:02,702][INFO ][o.e.p.PluginsService [2019-02-03T19:43:02,702][INFO ][o.e.p.PluginsService [2019-02-03T19:43:02,703][INFO ][o.e.p.PluginsService [2019-02-03T19:43:02,703][INFO ][o.e.p.PluginsService [2019-02-03T19:43:02,703][INFO ][o.e.p.PluginsService [2019-02-03T19:43:02,703][INFO ][o.e.p.PluginsService [2019-02-03T19:43:02,703][INFO ][o.e.p.PluginsService [2019-02-03T19:43:02,703][INFO ][o.e.p.PluginsService [2019-02-03T19:43:02,703][INFO ][o.e.p.PluginsService [2019-02-03T19:43:02,703][INFO ][o.e.p.PluginsService [2019-02-03T19:43:02,703][INFO ][o.e.p.PluginsService [2019-02-03T19:43:02,703][INFO ][o.e.p.PluginsService [2019-02-03T19:43:02,703][INFO ][o.e.p.PluginsService [2019-02-03T19:43:02,703][INFO ][o.e.p.PluginsService [2019-02-03T19:43:02,704][INFO ][o.e.p.PluginsService [2019-02-03T19:43:07,244][INFO ][o.e.x.s.a.s.FileRolesStore] [2019-02-03T19:43:07,903][INFO ][o.e.x.m.j.p.l.CppLog [2019-02-03T19:43:08,655][DEBUG][o.e.a.ActionModule [2019-02-03T19:43:09,021][INFO ][o.e.d.DiscoveryModule [2019-02-03T19:43:10,390][INFO ][o.e.n.Node [2019-02-03T19:43:10,390][INFO ][o.e.n.Node [2019-02-03T19:43:10,579][INFO ][o.e.t.TransportService [2019-02-03T19:43:13,654][INFO ][o.e.c.s.MasterService [2019-02-03T19:43:13,659][INFO ][o.e.c.s.ClusterApplierService] [2019-02-03T19:43:13,692][INFO ][o.e.x.s.t.n.Security [2019-02-03T19:43:13,693][INFO ][o.e.n.Node [2019-02-03T19:43:13,730][WARN ][o.e.x.s.a.s.m.Native [2019-02-03T19:43:13,806][INFO ][o.e.g.GatewayService ] [JrSNOHN] using [1] data paths, mounts [[/ (/dev/nvme0n1p1)]], net usable_space [5.4gb], net total_space [9.6gb], types [ext4]
] [JrSNOHN] heap size [1007.3mb], compressed ordinary object pointers [true]
] [JrSNOHN] node name derived from node ID [JrSNOHNgQh2FawKwALtK8g]; set [node.name] to override
] [JrSNOHN] version[6.5.4], pid[5252], build[default/deb/d2ef93d/2018-12-17T21:17:40.758843Z], OS[Linux/4.4.0-1074-aws/amd64], JVM[Or$
] [JrSNOHN] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccup$
] [JrSNOHN] loaded module [aggs-matrix-stats]
] [JrSNOHN] loaded module [analysis-common]
] [JrSNOHN] loaded module [ingest-common]
] [JrSNOHN] loaded module [lang-expression]
] [JrSNOHN] loaded module [lang-mustache]
] [JrSNOHN] loaded module [lang-painless]
] [JrSNOHN] loaded module [mapper-extras]
] [JrSNOHN] loaded module [parent-join]
] [JrSNOHN] loaded module [percolator]
] [JrSNOHN] loaded module [rank-eval]
] [JrSNOHN] loaded module [reindex]
] [JrSNOHN] loaded module [repository-url]
] [JrSNOHN] loaded module [transport-netty4]
] [JrSNOHN] loaded module [tribe]
] [JrSNOHN] loaded module [x-pack-ccr]
] [JrSNOHN] loaded module [x-pack-core]
] [JrSNOHN] loaded module [x-pack-deprecation]
] [JrSNOHN] loaded module [x-pack-graph]
] [JrSNOHN] loaded module [x-pack-logstash]
] [JrSNOHN] loaded module [x-pack-ml]
] [JrSNOHN] loaded module [x-pack-monitoring]
] [JrSNOHN] loaded module [x-pack-rollup]
] [JrSNOHN] loaded module [x-pack-security]
] [JrSNOHN] loaded module [x-pack-sql]
] [JrSNOHN] loaded module [x-pack-upgrade]
] [JrSNOHN] loaded module [x-pack-watcher]
] [JrSNOHN] no plugins loaded
[JrSNOHN] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
MessageHandler] [JrSNOHN] [controller/5308] [Main.cc@109] controller (64 bit): Version 6.5.4 (Build b616085ef32393) Copyright (c) 2018 Ela$
] [JrSNOHN] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
] [JrSNOHN] using discovery type [zen] and host providers [settings]
] [JrSNOHN] initialized
] [JrSNOHN] starting ...
] [JrSNOHN] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
] [JrSNOHN] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {JrSNOHN}{JrSNOHNgQh2FawKwALtK8g}{NoJJhAXkRXimqfku6Ki9$
[JrSNOHN] new_master {JrSNOHN}{JrSNOHNgQh2FawKwALtK8g}{NoJJhAXkRXimqfku6Ki9SQ}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=205$
Netty4HttpServerTransport] [JrSNOHN] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
] [JrSNOHN] started
RoleMappingStore] [JrSNOHN] Failed to clear cache for realms []
] [JrSNOHN] recovered [0] indices into cluster_state

Arvind_Singh · February 3, 2019, 9:51pm

Below one the latest logs-

[2019-02-03T20:57:49,245][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [percolator]
[2019-02-03T20:57:49,245][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [rank-eval]
[2019-02-03T20:57:49,245][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [reindex]
[2019-02-03T20:57:49,245][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [repository-url]
[2019-02-03T20:57:49,245][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [transport-netty4]
[2019-02-03T20:57:49,245][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [tribe]
[2019-02-03T20:57:49,245][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-ccr]
[2019-02-03T20:57:49,245][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-core]
[2019-02-03T20:57:49,245][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-deprecation]
[2019-02-03T20:57:49,245][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-graph]
[2019-02-03T20:57:49,246][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-logstash]
[2019-02-03T20:57:49,246][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-ml]
[2019-02-03T20:57:49,246][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-monitoring]
[2019-02-03T20:57:49,246][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-rollup]
[2019-02-03T20:57:49,246][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-security]
[2019-02-03T20:57:49,246][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-sql]
[2019-02-03T20:57:49,246][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-upgrade]
[2019-02-03T20:57:49,246][INFO ][o.e.p.PluginsService ] [JrSNOHN] loaded module [x-pack-watcher]
[2019-02-03T20:57:49,246][INFO ][o.e.p.PluginsService ] [JrSNOHN] no plugins loaded
[2019-02-03T20:57:53,794][INFO ][o.e.x.s.a.s.FileRolesStore] [JrSNOHN] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2019-02-03T20:57:54,714][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [JrSNOHN] [controller/1538] [Main.cc@109] controller (64 bit): Version 6.5.4 (Build b616085ef32393) Copyright (c) 2018 Ela$
[2019-02-03T20:57:55,236][DEBUG][o.e.a.ActionModule ] [JrSNOHN] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2019-02-03T20:57:55,471][INFO ][o.e.d.DiscoveryModule ] [JrSNOHN] using discovery type [zen] and host providers [settings]
[2019-02-03T20:57:56,292][INFO ][o.e.n.Node ] [JrSNOHN] initialized
[2019-02-03T20:57:56,293][INFO ][o.e.n.Node ] [JrSNOHN] starting ...
[2019-02-03T20:57:56,470][INFO ][o.e.t.TransportService ] [JrSNOHN] publish_address {172.31.14.91:9300}, bound_addresses {[::]:9300}
[2019-02-03T20:57:56,490][INFO ][o.e.b.BootstrapChecks ] [JrSNOHN] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-02-03T20:57:59,593][INFO ][o.e.c.s.MasterService ] [JrSNOHN] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {JrSNOHN}{JrSNOHNgQh2FawKwALtK8g}{KRaGeqxcT4aPx2Y-Ec9g$
[2019-02-03T20:57:59,599][INFO ][o.e.c.s.ClusterApplierService] [JrSNOHN] new_master {JrSNOHN}{JrSNOHNgQh2FawKwALtK8g}{KRaGeqxcT4aPx2Y-Ec9gxg}{172.31.14.91}{172.31.14.91:9300}{ml.machine_memo$
[2019-02-03T20:57:59,634][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [JrSNOHN] publish_address {172.31.14.91:9200}, bound_addresses {[::]:9200}
[2019-02-03T20:57:59,635][INFO ][o.e.n.Node ] [JrSNOHN] started
[2019-02-03T20:57:59,946][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [JrSNOHN] Failed to clear cache for realms []
[2019-02-03T20:58:00,006][INFO ][o.e.l.LicenseService ] [JrSNOHN] license [c685a2d0-5bff-49d0-94fc-a5ae78da5162] mode [basic] - valid
[2019-02-03T20:58:00,023][INFO ][o.e.g.GatewayService ] [JrSNOHN] recovered [0] indices into cluster_state

Arvind_Singh · February 3, 2019, 9:57pm

When i curl using private ip i get correct output-

{
"name" : "JrSNOHN",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "bfdy4EQLQWWCJDkx3JT80w",
"version" : {
"number" : "6.5.4",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "d2ef93d",
"build_date" : "2018-12-17T21:17:40.758843Z",
"build_snapshot" : false,
"lucene_version" : "7.5.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}

But when i curl using public ip this what i get-

Failed to connect to 13.232.169.223 port 9200: Connection timed out

warkolm · February 3, 2019, 9:59pm

It's still binding to localhost, so perhaps your config changes are not correct?

Also, please format logs/config/output using the </> button, or markdown-style backticks. It makes it much easier to read.

Arvind_Singh · February 3, 2019, 10:17pm

No that's not the case if u see my last logs, you will see it is not bound to localhost- I had changed network.host to 0.0.0.0

JrSNOHN] publish_address {172.31.14.91:9200}, bound_addresses {[::]:9200}
[2019-02-03T20:57:59,635][INFO ][o.e.n.Node ] [JrSNOHN] started
[2019-02-03T20:57:59,946][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [JrSNOHN] Failed to clear cache for realms
[2019-02-03T20:58:00,006][INFO ][o.e.l.LicenseService ] [JrSNOHN] license [c685a2d0-5bff-49d0-94fc-a5ae78da5162] mode [basic] - valid
[2019-02-03T20:58:00,023][INFO ][o.e.g.GatewayService ] [

warkolm · February 3, 2019, 10:22pm

You have two different IPs?

Arvind_Singh · February 3, 2019, 10:34pm

Yes public and private IP. I can connect to private IP but not to public IP.

warkolm · February 3, 2019, 10:35pm

Right, cause Elasticsearch is not binding to the public IP. So what did you actually set in the config.

I hope you have access control on this cluster by the way. Exposing things to the internet is not a good idea.

Arvind_Singh · February 3, 2019, 10:40pm

Thanks for your concern. this is just for my test server..

But how is that possible possible because I have another server where it running perfectly fine having same configuration in .yml file of elasticsearh.

warkolm · February 3, 2019, 10:41pm

That's not really the point. It's bad practise.

It'd help if you posted our your config.
Please make sure you use the code format button, as mentioned above.

Arvind_Singh · February 3, 2019, 10:42pm

Just the default settings I have in .yml file except network.host points to 0.0.0.0. rest are untouched. I also added http cors setting there but that does not have any effect.

Arvind_Singh · February 3, 2019, 10:51pm

#cluster.name: my-application
#node.name: node-1

#node.attr.rack: r1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
#bootstrap.memory_lock: true
network.host: 0.0.0.0
#http.port: 9200
#discovery.zen.ping.unicast.hosts: ["host1", "host2"]
#discovery.zen.minimum_master_nodes:
# Block initial recovery after a full cluster restart until N nodes are started:
#gateway.recover_after_nodes: 3
#action.destructive_requires_name: true
http.enabled: true
http.cors.enabled: true
http.cors.allow-origin: "*"

warkolm · February 3, 2019, 10:52pm

And what IPs are attached to the network interfaces on the host?

Arvind_Singh · February 3, 2019, 10:54pm

172.31.14.91- ens5
:127.0.0.1 -lo

i can see these two in ifconfig

warkolm · February 3, 2019, 10:55pm

Ok, then it cannot bind to that public IP if it's not attached to an interface on the host. So you will need to figure that part out.

Arvind_Singh · February 3, 2019, 11:01pm

but that is the same case with my another server. i can see same network interfaces with private ip of ens5 type and loopback of lo type there as well. no public ip attached there. but can connect to elasticsearch without any issue in that server