I am new to ELK and I spent days to troubleshoot and still not able to find out what's wrong with my configuration.
The following is my problem and what I have done to troubleshoot
-
I setup a basic ELK 7.1 on a single host
-
logstash can accept various kinds of log from localhost, but it fails to accept logs from remote machines
-
I troubleshoot by changing a very basic logstash.conf.
-
the following was my first logstash.conf that accepted stdin and it worked
input { stdin { } }
output { elasticsearch { hosts => ["localhost:9200"] } } -
I modified the logstash.conf to the following
input { tcp { port => 5017 } }
output { elasticsearch { hosts => ["localhost:9200"] } } -
I changed my router to write syslog to my logstash IP and port 5017, nothing is captured in logstash
-
I tried using netcat from another linux using the following command and the command neither timeout nor finish
echo "message" | nc -q0 192.168.1.200 5017 -
I tested the same netcat command from localhost and it worked.
-
I could find logstash listen the right port from log
[2019-05-26T03:54:12,430][INFO ][logstash.inputs.tcp ] Starting tcp input listener {:address=>"0.0.0.0:5017", :ssl_enable=>"false"}
[2019-05-26T03:54:12,618][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>} -
I ran "netstat -a | grep 5017" and I confirmed the host was listening
tcp6 0 0 [::]:5017 [::]:* LISTEN -
I ran "tcpdump tcp port 5017" and it received some data from the remote netcat command
listening on ens32, link-type EN10MB (Ethernet), capture size 262144 bytes
04:40:07.647365 IP 192.168.1.73.41174 > 192.168.1.200.5017: Flags [S], seq 3625096655, win 29200, options [mss 1460,sackOK,TS val 1674425657 ecr 0,nop,wscale 6], length 0
What have I missed ? Any help is appreciated.