Date format for YYYY-mm-dd HH:mm:ss,SSS?

Elastic Stack Logstash
#1

I looked at the Elasticsearch date format docs after reading this post and can't find the format for a date of the form YYYY-mm-dd HH:mm:ss,SSSS, e.g., 2017-03-29 10:00:00,123, where there's a comma for the milliseconds part. The log looks like

2017-03-29 10:00:00,123 INFO [com.company.app] This is a log

and my Grok filter to get the log date is

filter {
  if [type] == "artim-learning" {
    grok {
      match => {
        "message" => [
          "%{TIMESTAMP_ISO8601:logdate} ....other fields..."
        }
      }
    }
  }
}

The default Logstash mapping maps "logdate" as a string. If I want to map it as a date, what format do i use?

Thanks.

1 Like
Grok filter TIMESTAMP_ISO8601 appears in Kibana as String
#2

Normally one uses the date filter to parse a timestamp and produce an ISO8601 timestamp that Elasticsearch automatically treats as a date.

#3

So like this?

filter {
  if [type] == "artim-learning" {
    grok {
      match => {
        "message" => [
          "%{TIMESTAMP_ISO8601:logdate} ....other fields..."
        }
      }
    }
    date {
      match => [ "logdate", "YYYY-mm-dd HH:mm:ss,SSSS" ]
    }
  }
}

Thanks!

#4

Close: YYYY-MM-dd HH:mm:ss,SSS

1 Like
#5

@magnusbaeck, thanks!

#6

I added the date filter, but I still see this when I query the mapping

date {
  match => [ "logdate", "YYYY-MM-dd HH:mm:ss,SSS" ]
}

      "logdate" : {
        "type" : "string",
        "norms" : {
          "enabled" : false
        },
        "fielddata" : {
          "format" : "disabled"
        },
        "fields" : {
          "raw" : {
            "type" : "string",
            "index" : "not_analyzed",
            "ignore_above" : 256
          }
        }
      },

One discrepancy is the docs show the year as lowercase "y" vs. uppercase "Y" for the year?

#7

I added the date filter, but I still see this when I query the mapping

Did you create a new index? Mappings of existing indexes can't be changed.

One discrepancy is the docs show the year as lowercase "y" vs. uppercase "Y" for the year?

Either will work.

#8

I create a new daily index, yes.

#9

The date filter by defaults put the parsed timestamp in the default @timestamp field.

#10

@Christian_Dahlqvist, how can I ADDITIONALY put the date filter's parsed timestamp in my logdate field?

#11

@Christian_Dahlqvist, I got it to work with this filter. Your hint about @timestamp being the default gave me the idea to set the target. Thanks!

date {
  match => [ "logdate", "YYYY-MM-dd HH:mm:ss,SSS" ]
  target => "logdate"
}
2 Likes
#12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.