Connectivity issues to Elasticsearch via tcp\9200

littlejob · July 23, 2019, 7:02pm

Testing out winlogbeat to send syslogs to elastic instance with a kibana front end.

kibana status page shows all things are green

two network options set in elasticsearch.yml are:
network.host: 127.0.0.1
http.port: 9200

I have also tried changing network.host to 0.0.0.0 - but then kibana complains it cannot access the elastic instance..

both instances are on the same box... if that was not already implied..

i see local connections established, but only the ipv6 address/interface as listening..

i believe due to this, when i curl the host:9200 i get a connection refused message..
curl: (7) Failed to connect to 10.68.70.151 port 9200: Connection refused

i can curl localhost:9200 on the box itself, works fine, so i know the service is running.

Firewall is also disabled for testing..

due to this the winlogbeat service cannot talk to elastic instance.. how do i force it to listen on ipv4? what am i missing? banging my head on this one...

lieutenganger · July 24, 2019, 9:49am

I have exactly the same problem.

No matter what value i set on elasticsearch.yml for network.host: entry, (except uncomment or localhost), the IP_HOST:9200 is unreachable even from the ELK server itself.

After hours looking for a response, all i found ends with someone saying "is a proxy issue", "firewall issue","network issue"… but i'm pretty sure that's not the problem because kibana web on port 5601 is working.

Please there is other place on the system where check config? is in fact imposible to work with ELK and Wazuh Server on separate hosts?

dgonzalezp · July 24, 2019, 11:25am

Hi @littlejob
Could you please share your elasticsearch.yml content file?

DavidTurner · July 25, 2019, 8:34am

network.host: 0.0.0.0 (or a more specific address) is the right way to do that. But you say that you tried this and it didn't work? Can you try again, but this time capture the logs from Elasticsearch and share them all here?

littlejob · July 25, 2019, 11:05am

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

network.host: 127.0.0.1
http.port: 9200

dgonzalezp · July 25, 2019, 11:12am

Try this:

In elasticsearch.yml go to the section "Discovery" then uncomment or add the next line:
discovery.seed_hosts: ["IP where winlogbeat is installed"]

Restart elasticsearch service

(if doesnt work try network.host: 0.0.0.0 with the previous configuration)

littlejob · July 25, 2019, 11:26am

discovery seed host did not warrant successful results.

network.host 0.0.0.0 results in a connection refused error when navigating to the site.

current state:
elasticsearch.yml
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200

kibana.yml
server.port: 5601
server.host: localhost (tried changing this to ipv4 address assigned to host - no luck)
elasticsearch.hosts: ["http://localhost:9200"] (tried changing this as well to ipv4 address, no luck)

DavidTurner · July 25, 2019, 11:28am

You need to look at the Elasticsearch logs to work out why network.host: 0.0.0.0 isn't helping. Share them here if you need help interpreting them.

dgonzalezp · July 25, 2019, 11:37am

Well then as @DavidTurner said you should check the logs. To do it easy way is with the command:

tail -f /var/log/elasticsearch/elasticsearch.log

You have to look for error messages but you can share a screenshot if you need help.

littlejob · July 25, 2019, 11:43am

[discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] hmm

dgonzalezp · July 25, 2019, 11:49am

Ok
Add again the line that I told before (discovery.seed_hosts: ["x.x.x.x"])
dont forget the "" and restart the elasticsearch service.
Then check the logs again.

DavidTurner · July 25, 2019, 12:05pm

Please don't post images of text. They are not exposed to searches, so other people having the same problem won't find this thread. They are also inaccessible to those of us using screenreaders.

The logs you shared describe precisely what you must do.

littlejob · July 25, 2019, 12:06pm

still getting connection refused to even get to kibana front end.

DavidTurner · July 25, 2019, 12:17pm

Again, please don't post images of text. I basically can't read it, it's so small. Please edit your post and replace the screenshot with properly-formatted text (using the </> button), so it looks like this:

[2019-07-24T18:32:21,546][INFO ][o.e.e.NodeEnvironment    ] [node-0] using [1] data paths, mounts [[/ (/dev/disk1s1)]], net usable_space [314.1gb], net total_space [465.6gb], types [apfs]
[2019-07-24T18:32:21,550][INFO ][o.e.e.NodeEnvironment    ] [node-1] using [1] data paths, mounts [[/ (/dev/disk1s1)]], net usable_space [314.1gb], net total_space [465.6gb], types [apfs]
[2019-07-24T18:32:21,554][INFO ][o.e.e.NodeEnvironment    ] [node-0] heap size [989.8mb], compressed ordinary object pointers [true]
[2019-07-24T18:32:21,555][INFO ][o.e.e.NodeEnvironment    ] [node-1] heap size [989.8mb], compressed ordinary object pointers [true]
[2019-07-24T18:32:21,558][INFO ][o.e.n.Node               ] [node-1] node name [node-1], node ID [5w45d9i9Sk2dAu2JK_SS4g], cluster name [elasticsearch]
[2019-07-24T18:32:21,559][INFO ][o.e.n.Node               ] [node-1] version[7.2.0], pid[29284], build[default/tar/508c38a/2019-06-20T15:54:18.811730Z], OS[Mac OS X/10.14.6/x86_64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12/12+33]
[2019-07-24T18:32:21,559][INFO ][o.e.n.Node               ] [node-1] JVM home [/Users/davidturner/jvms/jdk-12.jdk/Contents/Home]
[2019-07-24T18:32:21,558][INFO ][o.e.n.Node               ] [node-0] node name [node-0], node ID [Lq0apxi2QOqXD5h3TlXs6w], cluster name [elasticsearch]
[2019-07-24T18:32:21,560][INFO ][o.e.n.Node               ] [node-0] version[7.2.0], pid[29283], build[default/tar/508c38a/2019-06-20T15:54:18.811730Z], OS[Mac OS X/10.14.6/x86_64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12/12+33]
[2019-07-24T18:32:21,560][INFO ][o.e.n.Node               ] [node-1] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/var/folders/3f/0f8prhv16qx7l9vcpsx7d0y00000gn/T/elasticsearch-15961517705110120031, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Dio.netty.allocator.type=unpooled, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/Users/davidturner/discuss/191913/elasticsearch-7.2.0, -Des.path.conf=/Users/davidturner/discuss/191913/elasticsearch-7.2.0/config-1, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2019-07-24T18:32:21,561][INFO ][o.e.n.Node               ] [node-0] JVM home [/Users/davidturner/jvms/jdk-12.jdk/Contents/Home]
[2019-07-24T18:32:21,562][INFO ][o.e.n.Node               ] [node-0] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/var/folders/3f/0f8prhv16qx7l9vcpsx7d0y00000gn/T/elasticsearch-2740889428437936081, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Dio.netty.allocator.type=unpooled, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/Users/davidturner/discuss/191913/elasticsearch-7.2.0, -Des.path.conf=/Users/davidturner/discuss/191913/elasticsearch-7.2.0/config-0, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2019-07-24T18:32:23,369][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [aggs-matrix-stats]
[2019-07-24T18:32:23,370][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [analysis-common]
[2019-07-24T18:32:23,371][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [data-frame]
[2019-07-24T18:32:23,371][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [ingest-common]

dgonzalezp · July 25, 2019, 12:29pm

I havent seen any error in that log... are you sure that is refusing the connection?
And Im agree with @DavidTurner is better send text than screenshot, my bad.

littlejob · July 25, 2019, 1:15pm

my apologies, please see the below tailed log results

root@mothership:/etc/elasticsearch# tail -f /var/log/elasticsearch/elasticsearch.log
[2019-07-25T12:04:29,460][INFO ][o.e.b.BootstrapChecks    ] [mothership] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-07-25T12:04:29,491][INFO ][o.e.c.c.Coordinator      ] [mothership] cluster UUID [PpDDbUeUQo2LjnUphiJM-w]
[2019-07-25T12:04:29,589][INFO ][o.e.c.s.MasterService    ] [mothership] elected-as-master ([1] nodes joined)[{mothership}{AJTAdrVIQ2mYStSJt4lDqQ}{fTxVY3lVRM6bZrMa6UKt4A}{10.68.70.151}{10.68.70.151:9300}{ml.machine_memory=16455876608, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 6, version: 95, reason: master node changed {previous [], current [{mothership}{AJTAdrVIQ2mYStSJt4lDqQ}{fTxVY3lVRM6bZrMa6UKt4A}{10.68.70.151}{10.68.70.151:9300}{ml.machine_memory=16455876608, xpack.installed=true, ml.max_open_jobs=20}]}
[2019-07-25T12:04:29,755][INFO ][o.e.c.s.ClusterApplierService] [mothership] master node changed {previous [], current [{mothership}{AJTAdrVIQ2mYStSJt4lDqQ}{fTxVY3lVRM6bZrMa6UKt4A}{10.68.70.151}{10.68.70.151:9300}{ml.machine_memory=16455876608, xpack.installed=true, ml.max_open_jobs=20}]}, term: 6, version: 95, reason: Publication{term=6, version=95}
[2019-07-25T12:04:29,997][INFO ][o.e.h.AbstractHttpServerTransport] [mothership] publish_address {10.68.70.151:9200}, bound_addresses {[::]:9200}
[2019-07-25T12:04:29,998][INFO ][o.e.n.Node               ] [mothership] started
[2019-07-25T12:04:30,654][INFO ][o.e.l.LicenseService     ] [mothership] license [2c9dda8f-8420-410b-b308-fe1baaf4167c] mode [basic] - valid
[2019-07-25T12:04:30,666][INFO ][o.e.g.GatewayService     ] [mothership] recovered [4] indices into cluster_state
[2019-07-25T12:04:31,865][INFO ][o.e.c.r.a.AllocationService] [mothership] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[filebeat-7.2.0-2019.07.19-000001][0], [.kibana_1][0], [metricbeat-7.2.0-2019.07.19-000001][0]] ...]).
[2019-07-25T12:04:32,017][INFO ][o.e.c.m.MetaDataIndexTemplateService] [mothership] adding template [.management-beats] for index patterns [.management-beats]

then if i look for listening kibana port - i get the following:

root@mothership:/etc/elasticsearch# netstat -a -n | grep tcp | grep 5601
tcp        0      0 127.0.0.1:5601          0.0.0.0:*               LISTEN

but i am unable to browse to the front end page.. im going backwards now. .

kibana service logs show the following more recent logs.

Jul 25 12:04:23 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:23Z","tags":["warning","elasticsearch","admin"],"pid":15088,"message":"Unable to revive connection: http://localhost:9200/"}
Jul 25 12:04:23 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:23Z","tags":["warning","elasticsearch","admin"],"pid":15088,"message":"No living connections"}
Jul 25 12:04:23 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:23Z","tags":["warning","task_manager"],"pid":15088,"message":"PollError No Living connections"}
Jul 25 12:04:24 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:24Z","tags":["warning","elasticsearch","admin"],"pid":15088,"message":"Unable to revive connection: http://localhost:9200/"}
Jul 25 12:04:24 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:24Z","tags":["warning","elasticsearch","admin"],"pid":15088,"message":"No living connections"}
Jul 25 12:04:26 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:26Z","tags":["warning","elasticsearch","admin"],"pid":15088,"message":"Unable to revive connection: http://localhost:9200/"}
Jul 25 12:04:26 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:26Z","tags":["warning","elasticsearch","admin"],"pid":15088,"message":"No living connections"}
Jul 25 12:04:26 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:26Z","tags":["warning","task_manager"],"pid":15088,"message":"PollError No Living connections"}
Jul 25 12:04:26 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:26Z","tags":["warning","elasticsearch","admin"],"pid":15088,"message":"Unable to revive connection: http://localhost:9200/"}
Jul 25 12:04:26 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:26Z","tags":["warning","elasticsearch","admin"],"pid":15088,"message":"No living connections"}
Jul 25 12:04:29 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:29Z","tags":["warning","elasticsearch","admin"],"pid":15088,"message":"Unable to revive connection: http://localhost:9200/"}
Jul 25 12:04:29 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:29Z","tags":["warning","elasticsearch","admin"],"pid":15088,"message":"No living connections"}
Jul 25 12:04:29 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:29Z","tags":["warning","task_manager"],"pid":15088,"message":"PollError No Living connections"}
Jul 25 12:04:29 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:29Z","tags":["warning","elasticsearch","admin"],"pid":15088,"message":"Unable to revive connection: http://localhost:9200/"}
Jul 25 12:04:29 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:29Z","tags":["warning","elasticsearch","admin"],"pid":15088,"message":"No living connections"}
Jul 25 12:04:31 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:31Z","tags":["status","plugin:elasticsearch@7.2.0","info"],"pid":15088,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"No Living connections"}
Jul 25 12:04:32 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:32Z","tags":["license","info","xpack"],"pid":15088,"message":"Imported license information from Elasticsearch for the [data] cluster: mode: basic | status: active"}
Jul 25 12:04:32 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:32Z","tags":["status","plugin:xpack_main@7.2.0","info"],"pid":15088,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"No Living connections"}
Jul 25 12:04:32 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:32Z","tags":["status","plugin:graph@7.2.0","info"],"pid":15088,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"No Living connections"}
Jul 25 12:04:32 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:32Z","tags":}

DavidTurner · July 25, 2019, 2:22pm

Thanks, that's much better

Elasticsearch is listening on 10.68.70.151:9200:

It looks like it's working correctly now. However, Kibana is trying to talk to Elasticsearch on localhost:

I think you need to adjust Kibana to find Elasticsearch at its new home?

littlejob · July 25, 2019, 6:40pm

I appreciate the collective assistance here. After removing the localhost entries and defining my local server IP in both the elastic and kibana configs, this resolved my issue..

thanks again.

system · August 22, 2019, 6:40pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Unable to connect to Elasticsearch at http://localhost:9200 Elasticsearch	9	208693	July 5, 2017
Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to http://localhost:5601/api/status fails: fail to execute the HTTP GET request Beats winlogbeat	2	4069	January 12, 2022
Elasticsearch: Couldn't connect to any of the configured Elasticsearch hosts - connection refused] Elasticsearch	6	28961	January 11, 2020
Accessing elasticsearch api url Elasticsearch	2	36	August 2, 2024
Filebeat does not connect to elastcisearch Elastic Agent filebeat	1	143	April 30, 2024

Connectivity issues to Elasticsearch via tcp\9200

Related Topics