Logstash Date Filter not working on Kibana

Elastic Stack Logstash
1

date {
match => ["logdate", "YYYY.MM.dd HH:mm:ss,SSS"]
target => "logdate"
}

Above is my date filter configuration on my logstash config file. On kibana, when I create an index pattern, the only date option that shows up is @timestamp but not logdate.

2

What is the mapping of logdate? Have a sample of the data?

3

There is a logdate field which I believe parses correctly. Here is an example of the log time format: 2020-08-13 15:57:14,242

Here is my config file below:
    input {
      beats {
        port => "5044"
        host => "0.0.0.0"
        ssl => false
      }
    }
    filter {
      grok {
        match => [
          #*grok patterns*
        ]
      }
      date {
        match => ["logdate", "YYYY.MM.dd HH:mm:ss,SSS"]
        target => "logdate"
      }
    }
    output {
      elasticsearch {
        hosts => ["*hosts*"]
        index => "logdata-%{+YYYY.MM.dd}"
      }
      stdout{}
    }
4

What is the mapping for the field in elasticsearch? If is mapped as a string, then even if the date filter successfully parses it, elasticsearch will convert it from a Logstash::Timestamp to a string as it gets indexed.

5 
"logdate" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
6

OK, so it is mapped as text. You will need to start over with a new index, if the date filter is working it will get mapped as a date/time. You could use an index template to force this mapping but probably will not need to.

7

This is the 2nd index I have tried to use the date filter with, and in both cases logdate does not show up for time. How would I use the index template? I am not familiar with that.

8

It is documented here.

9

and your date is with -

2020-08-13 15:57:14,242

you need to change to YYYY-MM-dd

10

Yep that's it, sorry for the trouble!

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.