Test-netconnection to ELK-Server is failing on port 5044 from target machines

Hi Everyone, I have installed winlog beat and metric beat agents on 10 machines. The script I generated did not perform a test-connection to the ELK server I realized it later on so I ran this on the 10 machines individually and can see only 2 machines out of 10 can successfully test the connection on port 5044. Why the remaining 8 are failing? Also, I can only see the data from the two machines (that can successfully establish a connection to the ELK server) only for Sunday (installed the beat agents on sunday) and not even a log from yesterday (Monday). ELK server configurations looks good I do not know what is causing the issue. Any help is appreciated, thank you!

Can you actually make a connection from the source machine to the target machine's 5044 port?
You could use eg. telnet for that, on the source machine.
telnet <target machine's IP address> 5044
(Quiting telnet: ^], which is ctrl+], then enter quit and hit enter)

Thank you for the response @atira . I tried telnet and it keeps failing. Test Net-Connection also keeps failing. I checked all the configuration files and everything looks good. I do not know how to troubleshoot this issue.

If there are servers successfully connecting to the port, it means that the receiving server is okay.
There might be some network issue, eg. a firewall doesn't allow communication from the sender to the receiver's port, or SSL is not correctly set up etc.
It's hard to say without additional info. Did you check the beats' log, what do they say?

@atira I am attaching the winlog beat log file below:

I also looked into the memory consumption of the ELK server and logstash is using 100% (process)IDK what is causing the issue!

2018-04-19T13:38:47.625-0400 INFO instance/beat.go:468 Home path: [C:\Program Files\Beats\winlogbeat] Config path: [C:\Program Files\Beats\winlogbeat] Data path: [C:\ProgramData\winlogbeat] Logs path: [C:\ProgramData\winlogbeat\logs]
2018-04-19T13:38:47.627-0400 INFO instance/beat.go:475 Beat UUID: 8965b8a5-093d-4689-b453-8a8sfa84ag4q
2018-04-19T13:38:47.627-0400 INFO instance/beat.go:213 Setup Beat: winlogbeat; Version: 6.2.3
2018-04-19T13:38:47.627-0400 WARN instance/metrics_other.go:8 Metrics not implemented for this OS.
2018-04-19T13:38:47.628-0400 INFO pipeline/module.go:76 Beat name: TestSystem
2018-04-19T13:38:47.628-0400 INFO beater/winlogbeat.go:56 State will be read from and persisted to C:\ProgramData\winlogbeat.winlogbeat.yml
2018-04-19T13:38:47.629-0400 INFO instance/beat.go:301 winlogbeat start running.
2018-04-19T13:38:47.629-0400 INFO [monitoring] log/log.go:97 Starting metrics logging every 30s
2018-04-19T13:39:10.727-0400 ERROR pipeline/output.go:74 Failed to connect: dial tcp 10.1.10.102:5044: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
2018-04-19T13:39:17.631-0400 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"libbeat":{"config":{"module":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":1,"events":{"active":7,"published":7,"retry":7,"total":7}}},"msg_file_cache":{"SecurityHits":6,"SecurityMisses":1,"SecuritySize":1},"uptime":"{"server_time":"2018-04-19T17:39:17.6315296Z","start_time":"2018-04-19T17:38:47.615632Z","uptime":"30.0158976s","uptime_ms":"30015897"}"}}}

You have a connection failure to the Logstash host:port, but unfortunately the cause is unspecified. Winlogbeat just says it didn't receive an answer.
You'll have to solve this on OS/network level.
Open a powershell or cmd prompt on the Winlogbeat host and try this:

telnet 10.1.10.102 5044

I assume this will fail.

Finally, I am just doing some troubleshooting and nothing seems going to work. I found out there is some error with logstash. I am attaching the logs below pleaes help me how to troubleshoot this, thanks

● logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2018-04-18 19:20:53 EDT; 4 days ago
Main PID: 6096 (java)
CGroup: /system.slice/logstash.service
└─6096 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOu

Apr 18 20:06:21 elk-server logstash[6096]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Ruby-0-Thread-5: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7
Apr 18 20:15:13 elk-server logstash[6096]: Exception in thread "nioEventLoopGroup-3-2" java.lang.OutOfMemoryError: Java heap space
Apr 18 20:40:25 elk-server logstash[6096]: Exception in thread "defaultEventExecutorGroup-4-2" java.lang.OutOfMemoryError: Java heap space
Apr 18 20:49:55 elk-server logstash[6096]: Exception in thread "defaultEventExecutorGroup-4-1" java.lang.OutOfMemoryError: Java heap space
Apr 18 21:03:07 elk-server logstash[6096]: Exception in thread "Ruby-0-Thread-15: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/puma-2.16.0-java/lib/puma/thread_pool.rb:187" java.lang.NoClassDefFoundError: Could not initialize
Apr 18 21:04:06 elk-server logstash[6096]: at org.jruby.RubyThread.raise(RubyThread.java:921)
Apr 18 21:04:14 elk-server logstash[6096]: at org.jruby.RubyThread.raise(RubyThread.java:904)
Apr 18 21:04:19 elk-server logstash[6096]: at org.jruby.RubyThread.exceptionRaised(RubyThread.java:1222)
Apr 18 21:04:21 elk-server logstash[6096]: at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:124)
Apr 18 21:04:21 elk-server logstash[6096]: at java.lang.Thread.run(Thread.java:748)

This is a classic out of memory error.
It means you are working with larger objects that do not fit into Logstash JVM's memory.
Try to increase the heap size in the jvm.options file of Logstash.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.