Date format for YYYY-mm-dd HH:mm:ss,SSS?

Elastic Stack Logstash
1

I looked at the Elasticsearch date format docs after reading this post and can't find the format for a date of the form YYYY-mm-dd HH:mm:ss,SSSS, e.g., 2017-03-29 10:00:00,123, where there's a comma for the milliseconds part. The log looks like

2017-03-29 10:00:00,123 INFO [com.company.app] This is a log

and my Grok filter to get the log date is

filter {
  if [type] == "artim-learning" {
    grok {
      match => {
        "message" => [
          "%{TIMESTAMP_ISO8601:logdate} ....other fields..."
        }
      }
    }
  }
}

The default Logstash mapping maps "logdate" as a string. If I want to map it as a date, what format do i use?

Thanks.

1 Like
2

Normally one uses the date filter to parse a timestamp and produce an ISO8601 timestamp that Elasticsearch automatically treats as a date.

3

So like this?

filter {
  if [type] == "artim-learning" {
    grok {
      match => {
        "message" => [
          "%{TIMESTAMP_ISO8601:logdate} ....other fields..."
        }
      }
    }
    date {
      match => [ "logdate", "YYYY-mm-dd HH:mm:ss,SSSS" ]
    }
  }
}

Thanks!

4

Close: YYYY-MM-dd HH:mm:ss,SSS

2 Likes
5

@magnusbaeck, thanks!

6

I added the date filter, but I still see this when I query the mapping

date {
  match => [ "logdate", "YYYY-MM-dd HH:mm:ss,SSS" ]
}

      "logdate" : {
        "type" : "string",
        "norms" : {
          "enabled" : false
        },
        "fielddata" : {
          "format" : "disabled"
        },
        "fields" : {
          "raw" : {
            "type" : "string",
            "index" : "not_analyzed",
            "ignore_above" : 256
          }
        }
      },

One discrepancy is the docs show the year as lowercase "y" vs. uppercase "Y" for the year?

7

I added the date filter, but I still see this when I query the mapping

Did you create a new index? Mappings of existing indexes can't be changed.

One discrepancy is the docs show the year as lowercase "y" vs. uppercase "Y" for the year?

Either will work.

8

I create a new daily index, yes.

9

The date filter by defaults put the parsed timestamp in the default @timestamp field.

10

@Christian_Dahlqvist, how can I ADDITIONALY put the date filter's parsed timestamp in my logdate field?

11

@Christian_Dahlqvist, I got it to work with this filter. Your hint about @timestamp being the default gave me the idea to set the target. Thanks!

date {
  match => [ "logdate", "YYYY-MM-dd HH:mm:ss,SSS" ]
  target => "logdate"
}
3 Likes
12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.