Replacing @timestamp with log timestamp

[scayzer65 (simon cayzer)]

Hi

I have a Windows FTP Server and the FTP application is producing real-time logs. I would like to replace the default @timestamp with the timestamp from the log. The message filter is working fine but when i add the date filter to replace the @timestamp everything falls in a heap.

I am a complete novice when it comes to Grok patterns so I am reaching out to anyone who can give me a bit of guidance and explain what I am doing wrong.

Sample FTP log

08 Aug 2017 00:00:05	Default Site	10.xxx.xxx.xxx userid	LogOut	

The following timestamp pattern and filter works when no date filter has been added.

FTP_STAMP %{MONTHDAY} %{MONTH} %{YEAR} %{HOUR}:%{MINUTE}:%{SECOND}

input{
    beats{
        port => "5043"
    }
}

filter {
    if [type] == "log" {
        grok {
            patterns_dir => ["/etc/logstash/patterns"]
            match => { "message" => "%{FTP_STAMP:ftpstamp}%{DATA:LOGLEVEL}%{IPV4:client}%{SPACE}%{USERNAME:userid}%{SPACE}%{GREEDYDATA:message}" }
        }
        date {
           match => ["ftpstamp" , "dd MMM yyyy HH:mm:ss"]
           target => "@timestamp"
        }
    }
}

output{
    elasticsearch {
        hosts => ["10.xxx.xxx.xxx:9200"]
        index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
        document_type => "%{[@metadata][type]}"
    }
}

[magnusbaeck (Magnus Bäck)]

Please show an example event produced by this configuration. Copy/paste from Kibana's JSON tab.

[scayzer65 (simon cayzer)]

Hi

Sorry for the late reply but after a lot of head scratching I realised that the problem was not related to the grok pattern I submitted it was actually a fault with the Elasticsearch setup. It seemed that even simple grok patterns were failing to work that had previously been ok. I started from scratch by purging my setup from the server and re-installing Logstash, Elasticsearch and Kibana. I started filebeat on the windows server and added the filter I sent in the post and guess what? It worked.

Thanks once again

SC