Date filter - how to parse separately grokked DATE and TIME

Elastic Stack Logstash
1

Hello All,

I have separately grokked DATE and TIME fields from a comma-separated string.

I now wish to filter them from two string into a date/time format with the Logstash date filter plugin.

I'm struggling with the syntax. I've tried the following. Where am I going wrong?

Thx, Keith :^)

  • DATE groks successfully; format is 20191009
  • TIME groks successfully; format is 14:23:33

~ does not work, but doesn't break.

date {
    match => ["[DATE][TIME]", "yyyymmddHH:mm:ss"]
        target => "@timestamp"
    }

~ error: Invalid FieldReference: %{[DATE]}%{[TIME]}

    date {
        match => ["%{[DATE]}%{[TIME]}", "yyyymmddHH:mm:ss"]
        target => "@timestamp"
    }

EDIT: I suppose really what I am asking is how to concatenate the date and time. Using the "+" char, it would be:

date {
    match => ["[DATE]+[TIME]", "yyyymmddHH:mm:ss"]
        target => "@timestamp"
    }
2

For posterity, I used Ruby:

    ruby {
        code => "
            d = event.get('[msg][DATE]')
            t = event.get('[msg][TIME]')
            dt = d + '_'
            if t.kind_of? Array
                dt = dt + t[0]
            else
                dt = dt + t
            end
            event.set('date_time', dt)
        "
    }

date_time was then fed into the Date filter.

3

You could use a mutate filter to combine the fields into a single field, and then use the date filter targeting that field; in this case, I combined them into [@metadata][combined_date], and since it is a subkey of @metadata it will not be included in the output so there is no need to clean it up.

filter {
  mutate {
    update => {
      "[@metadata][combined_date]" => "%{[msg][DATE]} %{[msg][TIME]}"
    }
  }
  date {
    match => ["[@metadata][combined_date]", "yyyymmdd HH:mm:ss"]
    target => "@timestamp"
  }
}