Hi
I have a Windows FTP Server and the FTP application is producing real-time logs. I would like to replace the default @timestamp with the timestamp from the log. The message filter is working fine but when i add the date filter to replace the @timestamp everything falls in a heap.
I am a complete novice when it comes to Grok patterns so I am reaching out to anyone who can give me a bit of guidance and explain what I am doing wrong.
Sample FTP log
08 Aug 2017 00:00:05 Default Site 10.xxx.xxx.xxx userid LogOut
The following timestamp pattern and filter works when no date filter has been added.
FTP_STAMP %{MONTHDAY} %{MONTH} %{YEAR} %{HOUR}:%{MINUTE}:%{SECOND}
input{
beats{
port => "5043"
}
}
filter {
if [type] == "log" {
grok {
patterns_dir => ["/etc/logstash/patterns"]
match => { "message" => "%{FTP_STAMP:ftpstamp}%{DATA:LOGLEVEL}%{IPV4:client}%{SPACE}%{USERNAME:userid}%{SPACE}%{GREEDYDATA:message}" }
}
date {
match => ["ftpstamp" , "dd MMM yyyy HH:mm:ss"]
target => "@timestamp"
}
}
}
output{
elasticsearch {
hosts => ["10.xxx.xxx.xxx:9200"]
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}