Hello Team,
I need to put filters for logstash for querying data coming from FTP servers. I havent worked on filters much, just have rough idea, so i created one for some below test logs:
Thu Jul 4 06:01:45 2019 [pid 43249] [xyz] OK DOWNLOAD: Client \"x.x.x.x\", \"/commonupdater/sitestat.xml\", 118 bytes, 2.64Kbyte/sec","popId":"1","hostIpAddress":"x.x.x.x","host":"ftp-1-2","data_field":"raw","type":"ftp-log
Thu Jul 4 06:20:14 2019 [pid 55668] [xyz] OK DOWNLOAD: Client \"x.x.x.x\", \"/commonupdater/sitestat.xml\", 118 bytes, 2.58Kbyte/sec","popId":"2","hostIpAddress":"x.x.x.x","host":"ftp-2-2","data_field":"raw","type":"ftp-log"
Thu Jul 4 06:20:13 2019 [pid 55666] [xyz] OK LOGIN: Client \"x.x.x.x\", anon password \"NcFTP@\"","popId":"3","hostIpAddress":"x.x.x.x","host":"ftp-2-3","data_field":"raw","type":"ftp-log"
Thu Jul 4 06:20:13 2019 [pid 55667] CONNECT: Client \"x.x.x.x\"","popId":"4","hostIpAddress":"x.x.x.x","host":"ftp-1-2","data_field":"raw","type":"ftp-log"
Thu Jul 4 06:20:11 2019 [pid 43201] CONNECT: Client \"x.x.x.x\"","popId":"5","hostIpAddress":"x.x.x.x","host":"ftp-2-4","data_field":"raw","type":"ftp-log"
In these logs line I need following filters, rest can be ignored:
- Status: OK DOWNLOAD, FAIL DOWNLOAD, CONNECT
- Client IP
- File Name: /commonupdater/sitestat.xml, etc.
- Size of file: In bytes
- Download rate: In bytes/sec
For this I created fiter pattern:
filter {
grok {
match => {"message" => "%{MONTH} +%{MONTHDAY} %{TIME} %{YEAR} (\[%{GREEDYDATA:pidno}\] )?(\[%{WORD:comp}\] )?(%{WORD:status} )?(%{WORD:download}:)?(%{WORD:client} )?(\"%{IPV4:ipaddr}\", )?(\"%{GREEDYDATA:filename}\", )?(%{GREEDYDATA:size} )?(%{GREEDYDATA:speed} )?"}
}
mutate {
remove_field => [ "pidno", "comp", "download", "client" ]
}
}
Thanks in Advance