Simple Logstash Date Filter

Elastic Stack Logstash
1

I am parsing JSON, and one of my fields are:

"myDateTime":"20180510T042139.680 GMT"

Here is my Logstash Config:

input{
    file{
        type => "test"
        codec => json{ }
        path => "/home/my-dev/Documents/test/*.json"
        start_position => "beginning"
        sincedb_path => "/dev/null"
        close_older => 60
        max_open_files => 1000
    }
}

filter{
    json{ source => "message" }
    if [type] == "test" {
        date{ match => ["myDateTime","yyyyMMdd'T'HHmmss.SSS ZZZ"] }
    }
}

output{
    if [type] == "test" {
        stdout{ codec => rubydebug }
        elasticsearch{ 
              hosts => ["127.0.0.1:9200"]
              index => "test"
              codec => json
        }
    }
}

This is my Logstash stdout:

May 31 11:20:51 my-VirtualBox logstash[15734]: Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
May 31 11:21:02 my-VirtualBox logstash[15734]: {
May 31 11:21:02 my-VirtualBox logstash[15734]:     "AccountCode" => "BASIC",
May 31 11:21:02 my-VirtualBox logstash[15734]:      "myDateTime" => "20180510T042139.680 GMT",
May 31 11:21:02 my-VirtualBox logstash[15734]:        "@version" => "1",
May 31 11:21:02 my-VirtualBox logstash[15734]:            "type" => "test",
May 31 11:21:02 my-VirtualBox logstash[15734]:            "host" => "prismdev-VirtualBox",
May 31 11:21:02 my-VirtualBox logstash[15734]:            "path" => "/home/my-dev/Documents/test/test.json",
May 31 11:21:02 my-VirtualBox logstash[15734]:      "@timestamp" => 2018-05-10T04:21:39.680Z
May 31 11:21:02 my-VirtualBox logstash[15734]: }

But when I go to Kibana, I can only see @timestamp for a Time Filter when creating my Index Pattern. Any ideas on what I am doing wrong?

2

That is to be expected. The date filter successfully parses the myDateTime field and puts the result in @timestamp.

3

right, but Kibana doesn't recognize the "myDateTime" field as a date type of field, it is still a "string" rather than a "date".

4

Right, myDateTime is not a date. It is a string. You could use the target option on the date filter to overwrite myDateTime with a date, which (in a new index) should be a date.

5

That fixed it! Thanks :slight_smile:

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.