Winlogbeat -> Logstash -> ES failing to write

[cymlog]

I am am experimenting with Winlogbeat, Logstash, ES.
I am experiencing intermittent log writes into ES from logstash

I have configured a WinServer 2008 R2 Vm with Winlogbeat used the following config to logstash (ensuring the elasticsearch is commented out):
> output.logstash:
> # The Logstash hosts
> hosts: ["10.110.1.100:5044"]

Logstash server and ES are on an Ubuntu 16.04 server which i have checked is all running fine.
I have enabled logging on the Windows machine and I am seeing this:

2018-01-24T13:28:26Z ERR Failed to connect: dial tcp 10.110.1.100:5044: connectex: No connection could be made because the target machine actively refused it.

the target machine actively refused it.

I can ping and telnet to that logstash host. I have checked the host to see if the ports are open using netstat -ntlp:

tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 931/sshd
tcp 0 0 10.110.1.100:8888 0.0.0.0:* LISTEN 1321/python2.7
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 1040/nginx -g daemo
tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 774/node
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 1040/nginx -g daemo
tcp 0 0 127.0.0.1:61619 0.0.0.0:* LISTEN 1321/python2.7
tcp 0 0 0.0.0.0:61620 0.0.0.0:* LISTEN 1321/python2.7
tcp6 0 0 :::22 :::* LISTEN 931/sshd
tcp6 0 0 127.0.0.1:9600 :::* LISTEN 14065/java
tcp6 0 0 :::9200 :::* LISTEN 3254/java
tcp6 0 0 :::5044 :::* LISTEN 14065/java
tcp6 0 0 :::9300 :::* LISTEN 3254/java

After turning off all firewall and AV on the windows server this did not work. I attempted from another windows client the same thing but received an unable to connect.

I then tried turning logstash level to debug. I could see output being received:

output received {"event"=>{"computer_name"=>"computername.amadeupdomain.local", "process_id"=>560, "keywords"=>["Audit Success"]..........

So logstash is receiving the logs from winlogbeat

Logstash input:
input {
beats {
port => 5044
}
}

Logstash output:
output {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => false
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}

I have checked ES logs and this is what I am seeing there:
[2018-01-24T13:21:29,695][INFO ][o.e.c.m.MetaDataMappingService] [rIsMGDA] [winlogbeat-2015.04.18/vKyCEwizQSyycfhdlTyiQg] update_mapping [doc]

When searching Kibana->Discover for winlogstash-* I am not consistently seeing entries.
I am unsure if this is a client issue or logstash being unable to output to ES?

Can I have some further suggestions on how to see if the issue is between logstash and ES.

[JKhondhu (Jymit Singh Khondhu)]

Hi,

What event logs has Winlogbeat been configured to collect?
Review of those event logs, are they frequent? By this I am looking to figure if the event logging on the Windows server is in-line with what you are seeing coming in via Logstash or Elasticsearch.

Does the Ubuntu 16.04 server have UFW or iptables rate limiting incoming comms?

[cymlog]

winlogbeat 6.1.2
window log settings to:

logging.level: error
logging.to_files: true
logging.to_syslog: false
logging.files:
path: c:\ELK-Beats\logs
name: mybeat.log
keepfiles: 7

----------8< snip -----------------------
root@logs:/home/userficticious# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

--------8< Snip --------------------------------------------

user@logs:/home/user# sudo ufw status verbose
Status: inactive

There are no network firewall rules and as stated all client side firewall and AV has been disabled.

EDITED:
I am using get-content -path to 'tail' the events. And I am seeing
2018-01-24T14:32:41Z ERR Failed to publish events caused by: read tcp 10.110.1.31:63674->10.110.1.110:5044: i/o timeout

[cymlog]

Evidently I am not telling the truth....... it would appear that port 5044 is not open for business. What do I need to do to get logstash to listen on this port?

pretend@SERVER:~$ nmap -P0 10.110.1.100

Starting Nmap 7.01 ( https://nmap.org ) at 2018-01-24 15:05 GMT
Nmap scan report for 10.110.1.100
Host is up (0.00032s latency).
Not shown: 995 closed ports
PORT STATE SERVICE
443/tcp open https
8888/tcp open sun-answerbook
9200/tcp open wap-wsp

Nmap done: 1 IP address (1 host up) scanned in 13.05 seconds

[cymlog]

I can telnet from another Linux machine to logstash host but not windows clients - server 2008r2 or Win7. Is this related to the beats port 5044 being listed as IPv6?

[cymlog]

Winlogbeat is now logging to logstash on the 5044 port. I cannot figure what was preventing the connections on the network but it is working. Will take that!