Connectivity issues to Elasticsearch via tcp\9200

Elastic Stack Elasticsearch
1

Testing out winlogbeat to send syslogs to elastic instance with a kibana front end.

kibana status page shows all things are green

1
1.png1063×508 24.9 KB

two network options set in elasticsearch.yml are:
network.host: 127.0.0.1
http.port: 9200

I have also tried changing network.host to 0.0.0.0 - but then kibana complains it cannot access the elastic instance..

both instances are on the same box... if that was not already implied..

i see local connections established, but only the ipv6 address/interface as listening..
2

i believe due to this, when i curl the host:9200 i get a connection refused message..
curl: (7) Failed to connect to 10.68.70.151 port 9200: Connection refused

i can curl localhost:9200 on the box itself, works fine, so i know the service is running.

Firewall is also disabled for testing..

due to this the winlogbeat service cannot talk to elastic instance.. how do i force it to listen on ipv4? what am i missing? banging my head on this one...

1 Like
2

I have exactly the same problem.

No matter what value i set on elasticsearch.yml for network.host: entry, (except uncomment or localhost), the IP_HOST:9200 is unreachable even from the ELK server itself.

After hours looking for a response, all i found ends with someone saying "is a proxy issue", "firewall issue","network issue"… but i'm pretty sure that's not the problem because kibana web on port 5601 is working.

Please there is other place on the system where check config? is in fact imposible to work with ELK and Wazuh Server on separate hosts?

3

Hi @littlejob
Could you please share your elasticsearch.yml content file?

4

network.host: 0.0.0.0 (or a more specific address) is the right way to do that. But you say that you tried this and it didn't work? Can you try again, but this time capture the logs from Elasticsearch and share them all here?

5

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

network.host: 127.0.0.1
http.port: 9200

6

Try this:

In elasticsearch.yml go to the section "Discovery" then uncomment or add the next line:
discovery.seed_hosts: ["IP where winlogbeat is installed"]

Restart elasticsearch service

(if doesnt work try network.host: 0.0.0.0 with the previous configuration)

7

discovery seed host did not warrant successful results.

network.host 0.0.0.0 results in a connection refused error when navigating to the site.

current state:
elasticsearch.yml
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200

kibana.yml
server.port: 5601
server.host: localhost (tried changing this to ipv4 address assigned to host - no luck)
elasticsearch.hosts: ["http://localhost:9200"] (tried changing this as well to ipv4 address, no luck)

image

8

You need to look at the Elasticsearch logs to work out why network.host: 0.0.0.0 isn't helping. Share them here if you need help interpreting them.

9

Well then as @DavidTurner said you should check the logs. To do it easy way is with the command:

tail -f /var/log/elasticsearch/elasticsearch.log

You have to look for error messages but you can share a screenshot if you need help.

10

image
image.png1491×197 9.99 KB

[discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] hmm

11

Ok
Add again the line that I told before (discovery.seed_hosts: ["x.x.x.x"])
dont forget the "" and restart the elasticsearch service.
Then check the logs again.

12

Please don't post images of text. They are not exposed to searches, so other people having the same problem won't find this thread. They are also inaccessible to those of us using screenreaders.

The logs you shared describe precisely what you must do.

13

image
image.png1520×304 23.1 KB

still getting connection refused to even get to kibana front end.

14

Again, please don't post images of text. I basically can't read it, it's so small. Please edit your post and replace the screenshot with properly-formatted text (using the </> button), so it looks like this:

[2019-07-24T18:32:21,546][INFO ][o.e.e.NodeEnvironment    ] [node-0] using [1] data paths, mounts [[/ (/dev/disk1s1)]], net usable_space [314.1gb], net total_space [465.6gb], types [apfs]
[2019-07-24T18:32:21,550][INFO ][o.e.e.NodeEnvironment    ] [node-1] using [1] data paths, mounts [[/ (/dev/disk1s1)]], net usable_space [314.1gb], net total_space [465.6gb], types [apfs]
[2019-07-24T18:32:21,554][INFO ][o.e.e.NodeEnvironment    ] [node-0] heap size [989.8mb], compressed ordinary object pointers [true]
[2019-07-24T18:32:21,555][INFO ][o.e.e.NodeEnvironment    ] [node-1] heap size [989.8mb], compressed ordinary object pointers [true]
[2019-07-24T18:32:21,558][INFO ][o.e.n.Node               ] [node-1] node name [node-1], node ID [5w45d9i9Sk2dAu2JK_SS4g], cluster name [elasticsearch]
[2019-07-24T18:32:21,559][INFO ][o.e.n.Node               ] [node-1] version[7.2.0], pid[29284], build[default/tar/508c38a/2019-06-20T15:54:18.811730Z], OS[Mac OS X/10.14.6/x86_64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12/12+33]
[2019-07-24T18:32:21,559][INFO ][o.e.n.Node               ] [node-1] JVM home [/Users/davidturner/jvms/jdk-12.jdk/Contents/Home]
[2019-07-24T18:32:21,558][INFO ][o.e.n.Node               ] [node-0] node name [node-0], node ID [Lq0apxi2QOqXD5h3TlXs6w], cluster name [elasticsearch]
[2019-07-24T18:32:21,560][INFO ][o.e.n.Node               ] [node-0] version[7.2.0], pid[29283], build[default/tar/508c38a/2019-06-20T15:54:18.811730Z], OS[Mac OS X/10.14.6/x86_64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12/12+33]
[2019-07-24T18:32:21,560][INFO ][o.e.n.Node               ] [node-1] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/var/folders/3f/0f8prhv16qx7l9vcpsx7d0y00000gn/T/elasticsearch-15961517705110120031, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Dio.netty.allocator.type=unpooled, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/Users/davidturner/discuss/191913/elasticsearch-7.2.0, -Des.path.conf=/Users/davidturner/discuss/191913/elasticsearch-7.2.0/config-1, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2019-07-24T18:32:21,561][INFO ][o.e.n.Node               ] [node-0] JVM home [/Users/davidturner/jvms/jdk-12.jdk/Contents/Home]
[2019-07-24T18:32:21,562][INFO ][o.e.n.Node               ] [node-0] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/var/folders/3f/0f8prhv16qx7l9vcpsx7d0y00000gn/T/elasticsearch-2740889428437936081, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Dio.netty.allocator.type=unpooled, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/Users/davidturner/discuss/191913/elasticsearch-7.2.0, -Des.path.conf=/Users/davidturner/discuss/191913/elasticsearch-7.2.0/config-0, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2019-07-24T18:32:23,369][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [aggs-matrix-stats]
[2019-07-24T18:32:23,370][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [analysis-common]
[2019-07-24T18:32:23,371][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [data-frame]
[2019-07-24T18:32:23,371][INFO ][o.e.p.PluginsService     ] [node-0] loaded module [ingest-common]
15

I havent seen any error in that log... are you sure that is refusing the connection?
And Im agree with @DavidTurner is better send text than screenshot, my bad.

16

my apologies, please see the below tailed log results

root@mothership:/etc/elasticsearch# tail -f /var/log/elasticsearch/elasticsearch.log
[2019-07-25T12:04:29,460][INFO ][o.e.b.BootstrapChecks    ] [mothership] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-07-25T12:04:29,491][INFO ][o.e.c.c.Coordinator      ] [mothership] cluster UUID [PpDDbUeUQo2LjnUphiJM-w]
[2019-07-25T12:04:29,589][INFO ][o.e.c.s.MasterService    ] [mothership] elected-as-master ([1] nodes joined)[{mothership}{AJTAdrVIQ2mYStSJt4lDqQ}{fTxVY3lVRM6bZrMa6UKt4A}{10.68.70.151}{10.68.70.151:9300}{ml.machine_memory=16455876608, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 6, version: 95, reason: master node changed {previous [], current [{mothership}{AJTAdrVIQ2mYStSJt4lDqQ}{fTxVY3lVRM6bZrMa6UKt4A}{10.68.70.151}{10.68.70.151:9300}{ml.machine_memory=16455876608, xpack.installed=true, ml.max_open_jobs=20}]}
[2019-07-25T12:04:29,755][INFO ][o.e.c.s.ClusterApplierService] [mothership] master node changed {previous [], current [{mothership}{AJTAdrVIQ2mYStSJt4lDqQ}{fTxVY3lVRM6bZrMa6UKt4A}{10.68.70.151}{10.68.70.151:9300}{ml.machine_memory=16455876608, xpack.installed=true, ml.max_open_jobs=20}]}, term: 6, version: 95, reason: Publication{term=6, version=95}
[2019-07-25T12:04:29,997][INFO ][o.e.h.AbstractHttpServerTransport] [mothership] publish_address {10.68.70.151:9200}, bound_addresses {[::]:9200}
[2019-07-25T12:04:29,998][INFO ][o.e.n.Node               ] [mothership] started
[2019-07-25T12:04:30,654][INFO ][o.e.l.LicenseService     ] [mothership] license [2c9dda8f-8420-410b-b308-fe1baaf4167c] mode [basic] - valid
[2019-07-25T12:04:30,666][INFO ][o.e.g.GatewayService     ] [mothership] recovered [4] indices into cluster_state
[2019-07-25T12:04:31,865][INFO ][o.e.c.r.a.AllocationService] [mothership] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[filebeat-7.2.0-2019.07.19-000001][0], [.kibana_1][0], [metricbeat-7.2.0-2019.07.19-000001][0]] ...]).
[2019-07-25T12:04:32,017][INFO ][o.e.c.m.MetaDataIndexTemplateService] [mothership] adding template [.management-beats] for index patterns [.management-beats]

then if i look for listening kibana port - i get the following:

root@mothership:/etc/elasticsearch# netstat -a -n | grep tcp | grep 5601
tcp        0      0 127.0.0.1:5601          0.0.0.0:*               LISTEN

but i am unable to browse to the front end page.. im going backwards now. .

kibana service logs show the following more recent logs.

Jul 25 12:04:23 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:23Z","tags":["warning","elasticsearch","admin"],"pid":15088,"message":"Unable to revive connection: http://localhost:9200/"}
Jul 25 12:04:23 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:23Z","tags":["warning","elasticsearch","admin"],"pid":15088,"message":"No living connections"}
Jul 25 12:04:23 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:23Z","tags":["warning","task_manager"],"pid":15088,"message":"PollError No Living connections"}
Jul 25 12:04:24 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:24Z","tags":["warning","elasticsearch","admin"],"pid":15088,"message":"Unable to revive connection: http://localhost:9200/"}
Jul 25 12:04:24 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:24Z","tags":["warning","elasticsearch","admin"],"pid":15088,"message":"No living connections"}
Jul 25 12:04:26 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:26Z","tags":["warning","elasticsearch","admin"],"pid":15088,"message":"Unable to revive connection: http://localhost:9200/"}
Jul 25 12:04:26 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:26Z","tags":["warning","elasticsearch","admin"],"pid":15088,"message":"No living connections"}
Jul 25 12:04:26 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:26Z","tags":["warning","task_manager"],"pid":15088,"message":"PollError No Living connections"}
Jul 25 12:04:26 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:26Z","tags":["warning","elasticsearch","admin"],"pid":15088,"message":"Unable to revive connection: http://localhost:9200/"}
Jul 25 12:04:26 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:26Z","tags":["warning","elasticsearch","admin"],"pid":15088,"message":"No living connections"}
Jul 25 12:04:29 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:29Z","tags":["warning","elasticsearch","admin"],"pid":15088,"message":"Unable to revive connection: http://localhost:9200/"}
Jul 25 12:04:29 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:29Z","tags":["warning","elasticsearch","admin"],"pid":15088,"message":"No living connections"}
Jul 25 12:04:29 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:29Z","tags":["warning","task_manager"],"pid":15088,"message":"PollError No Living connections"}
Jul 25 12:04:29 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:29Z","tags":["warning","elasticsearch","admin"],"pid":15088,"message":"Unable to revive connection: http://localhost:9200/"}
Jul 25 12:04:29 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:29Z","tags":["warning","elasticsearch","admin"],"pid":15088,"message":"No living connections"}
Jul 25 12:04:31 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:31Z","tags":["status","plugin:elasticsearch@7.2.0","info"],"pid":15088,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"No Living connections"}
Jul 25 12:04:32 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:32Z","tags":["license","info","xpack"],"pid":15088,"message":"Imported license information from Elasticsearch for the [data] cluster: mode: basic | status: active"}
Jul 25 12:04:32 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:32Z","tags":["status","plugin:xpack_main@7.2.0","info"],"pid":15088,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"No Living connections"}
Jul 25 12:04:32 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:32Z","tags":["status","plugin:graph@7.2.0","info"],"pid":15088,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"No Living connections"}
Jul 25 12:04:32 mothership kibana[15088]: {"type":"log","@timestamp":"2019-07-25T12:04:32Z","tags":}
17

Thanks, that's much better

Elasticsearch is listening on 10.68.70.151:9200:

It looks like it's working correctly now. However, Kibana is trying to talk to Elasticsearch on localhost:

I think you need to adjust Kibana to find Elasticsearch at its new home?

18

I appreciate the collective assistance here. After removing the localhost entries and defining my local server IP in both the elastic and kibana configs, this resolved my issue..

thanks again.

1 Like
19

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.