I have two servers, getting logs from the same sources, parallel streams of netflow and syslog.
64Gb RAM, 8 cores, 500Gb SSD VMs running on ESXI 6.7.
I build the servers using Ansible and Cobbler so they are "identical".
I run diffs against their configs, ELK and Red Hat, identical
I have logstash listening on four ports, 2x netflow and 2x syslog
I can do tcpdumps and I see the incoming UDP messages
I check IPTABLES and the ports are allowed
One server 1 I can see all FOUR of logs being processed into elastic
One server 2 I can see only ONE logs being processed into elastic
Of the logs I am missing, I dont even see the logstash debug telling me that netflow has not recieved a template yet.
So I am guessing that the UDP messages are not getting to the input filter, or the input filter is not processing them
So pick one of the missing logs and strip my config down to just one, cisco netflow ....
input {
# Netflow from Cisco routers for BGAN only
udp {
port => 2100
type => "cisco"
add_field => { "[@metadata][fingerprint]" => "" }
add_field => { "IsFlow" => "True" }
add_field => { "subtype" => "netflow" }
codec => netflow {
versions => [1, 5, 9, 10]
}
}
}
filter {
}
output {
For debug only, CAREFUL LOG DIRECTORY DOES NOT FILL UP
if [type] == "cisco" {
file {
path => "/var/log/logstash/message-%{type}-%{+YYYY.MM.dd}.log"
codec => rubydebug
}
}
}
Starts ok ...
[2019-03-05T22:54:05,929][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2019-03-05T23:03:04,276][WARN ][logstash.runner ] SIGTERM received. Shutting down.
[2019-03-05T23:03:05,683][INFO ][logstash.pipeline ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x55d0a32a run>"}
[2019-03-05T23:03:51,160][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.5.3"}
[2019-03-05T23:03:57,458][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-03-05T23:03:57,567][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x1e0ff11f run>"}
[2019-03-05T23:03:57,605][INFO ][logstash.inputs.udp ] Starting UDP listener {:address=>"0.0.0.0:2100"}
[2019-03-05T23:03:57,640][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[2019-03-05T23:03:57,681][INFO ][logstash.inputs.udp ] UDP listener started {:address=>"0.0.0.0:2100", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
[2019-03-05T23:03:57,866][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
Port there
$ netstat -lnp | grep 2100
udp 0 0 0.0.0.0:2100 0.0.0.0:* 614/java
Firewall open
$ iptables -nL | grep 2100
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpts:2100:2103 ctstate NEW
Packets arriving
$ tcpdump -l -c 1 -nn -i eth0 -X udp port 2100 | head -7
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
23:02:12.333000 IP A.B.C.D.62979 > W.X.Y.Z.2100: UDP, length 1349
0x0000: 4500 0561 6e35 0000 fe11 1c59 0a6e 0f0b E..an5.....Y.n..
0x0010: 0a6e 0917 f603 0834 054d 5571 0009 0019 .n.....4.MUq....
0x0020: a34c 7c7c 5c7e fff4 0065 41ad 0000 0000 .L||~...eA.....
0x0030: 0108 0531 bcac c68f 0a1c 7c1c 0000 0000 ...1......|.....
0x0040: 0000 0014 1732 1776 0006 001d 1800 0000 .....2.v........
0x0050: 000a 6e47 0300 0000 4000 0000 01a3 4c3d ..nG....@.....L=
lets look for the log file ....
$ ls -l
total 108
-rw-r--r-- 1 logstash logstash 110002 Mar 5 23:03 logstash-plain.log
-rw-r--r-- 1 logstash logstash 0 Mar 5 13:28 logstash-slowlog-plain.log
Not there !
CPU and MEMORY ok
top - 23:11:53 up 8 min, 3 users, load average: 0.14, 0.15, 0.12
Tasks: 176 total, 2 running, 174 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.2 us, 0.1 sy, 0.6 ni, 99.1 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 65810316 total, 56674440 free, 7719880 used, 1415996 buff/cache
KiB Swap: 8388604 total, 8388604 free, 0 used. 57856488 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
601 logstash 39 19 9389564 1.078g 16476 S 5.6 1.7 2:01.07 java
1320 elastic+ 20 0 13.894g 5.595g 30328 S 1.3 8.9 0:53.01 java
disk ok
$ df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda3 1022282036 8744476 1013537560 1% /
devtmpfs 32895740 0 32895740 0% /dev
tmpfs 32905156 0 32905156 0% /dev/shm
tmpfs 32905156 8740 32896416 1% /run
tmpfs 32905156 0 32905156 0% /sys/fs/cgroup
/dev/sda7 10475520 1027972 9447548 10% /var/log
/dev/sda5 10475520 32928 10442592 1% /home
/dev/sda6 10475520 33072 10442448 1% /tmp
/dev/sda8 10475520 35044 10440476 1% /var/log/audit
/dev/sda2 520876 121080 399796 24% /boot
/dev/sda10 96050 0 96050 0% /boot/efi
tmpfs 6581032 0 6581032 0% /run/user/0