Logstash - The default timestamp field does not match my log field

[ozonshak]

Hello. I have an existing Elastic stack that is pulling in app and web server logs. For the web site, I have 2 software stacks - 1 is using an older apache format (comma separated values) and 1 is using a newer JSON format. With the JSON format, I'm trying to leverage my existing YAML filter and modifying it to try and pick up the JSON format. I've been able to get everything working except for the timestamp field. It is reflecting the upload time of the log, not the actual log entry.

Here is an example of an Apache log entry:

{"type":"ihs-access","datetime":"2024-02-15T23:59:59.999 EST","@version":"1","protocol":"HTTP/1.1","remoteHost":"10.10.10.10","port":443,"method":"GET","url":"/this/is/a/path/product/details/abcd","query":"","status":200,"bytes":1826,"elapsedTime":3344,"referer":"-","libertyServer":"serverhost:5443","tealeafId":"-","sessionId":"-","requestId":"-","jsessionId":"-","userAgent":"Apache-HttpClient/4.5.2 (Java/1.7.0)"}

Here is a snippet of my logstash pipeline:

input {
    beats {
        port => "5044"
    }
}
filter {
    if [fields][log_type] == "weblogs" {
 json {
          source => "message"
          tag_on_failure => [ "_jsonparsefailure" ]
       }
       mutate { replace => { type => "apache_access" } }
       mutate {
          rename => { "datetime" => "httpTime" }
          rename => { "remoteHost" => "httpHostIP" }
          rename => { "method" => "httpRequestMethod" }
          rename => { "url" => "httpRequest" }
          rename => { "query" => "httpRequestQueryParms" }
          rename => { "status" => "httpResponseCode" }
          rename => { "bytes" => "httpResponseSize" }
          rename => { "elapsedTime" => "httpResponseTime" }
          rename => { "referer" => "httpReferer" }
          rename => { "jsessionId" => "httpJSessionID" }
          rename => { "userAgent" => "httpUserAgent" }
          convert => { "httpResponseCode" => "integer" }
          convert => { "httpResponseSize" => "integer" }
          convert => { "httpResponseTime" => "integer" }
       }
       date {
          locale => "en"
          match => ["httpTime", "yyyy-MM-dd'T'HH:mm:ss:SSSz"]
          timezone => "US/Eastern"
       }
    }

I'm changing the field name from "datetime" to "httpTime" and then formatting the "httpTime" field.

I'm assuming from the above that my date/time format is wrong on the date match. Can someone please help me format this properly based upon my input format? Thanks!

[Badger]

Try match => ["httpTime", "yyyy-MM-dd'T'HH:mm:ss.SSS ZZZ"]

[ozonshak]

Nope, didn't work. Here is the modification:

date {
          locale => "en"
          match => ["httpTime", "yyyy-MM-dd'T'HH:mm:ss:SSS ZZZ"]
          timezone => "US/Eastern"
          #timezone => "US/Central"
       }

Here is the index snippet:

the @timestamp is still reflecting the time of the log ingestion.

[Badger]

It works for me

input { generator { count => 1 lines => [ '' ] } }

output { stdout { codec => rubydebug { metadata => false } } }
filter {
    mutate { remove_field => [ "event", "host", "xlog" ] }

    mutate { add_field => { "httpTime" => "2024-02-15T23:59:59.999 EST" } }
    date {
        locale => "en"
        match => ["httpTime", "yyyy-MM-dd'T'HH:mm:ss.SSS ZZZ"]
    }
}

produces

"@timestamp" => 2024-02-16T04:59:59.999Z,
  "httpTime" => "2024-02-15T23:59:59.999 EST",

Since the timezone is specified in [httpTime] the timezone option has no effect.

[ozonshak]

Any idea what else I can try? Not sure why your test works but mine is does not. I removed that timezone option, but the results are the same. I'm at a loss at this point.

[ozonshak]

Ugh. Figured out the final issue. My input vs joda match was off by a single character:

input:
"datetime":"2024-02-15T23:59:59.999 EST"

match (before):
match => ["httpTime", "yyyy-MM-dd'T'HH:mm:ss:SSS ZZZ"]

match (after):
match => ["httpTime", "yyyy-MM-dd'T'HH:mm:ss.SSS ZZZ"]

I had a colon right before the SSS and it should have been a dot. Gheesh! That was easy to miss.

@Badger Thanks for all your help narrowing this down!