Hello. I have an existing Elastic stack that is pulling in app and web server logs. For the web site, I have 2 software stacks - 1 is using an older apache format (comma separated values) and 1 is using a newer JSON format. With the JSON format, I'm trying to leverage my existing YAML filter and modifying it to try and pick up the JSON format. I've been able to get everything working except for the timestamp field. It is reflecting the upload time of the log, not the actual log entry.
Here is an example of an Apache log entry:
{"type":"ihs-access","datetime":"2024-02-15T23:59:59.999 EST","@version":"1","protocol":"HTTP/1.1","remoteHost":"10.10.10.10","port":443,"method":"GET","url":"/this/is/a/path/product/details/abcd","query":"","status":200,"bytes":1826,"elapsedTime":3344,"referer":"-","libertyServer":"serverhost:5443","tealeafId":"-","sessionId":"-","requestId":"-","jsessionId":"-","userAgent":"Apache-HttpClient/4.5.2 (Java/1.7.0)"}
Here is a snippet of my logstash pipeline:
input {
beats {
port => "5044"
}
}
filter {
if [fields][log_type] == "weblogs" {
json {
source => "message"
tag_on_failure => [ "_jsonparsefailure" ]
}
mutate { replace => { type => "apache_access" } }
mutate {
rename => { "datetime" => "httpTime" }
rename => { "remoteHost" => "httpHostIP" }
rename => { "method" => "httpRequestMethod" }
rename => { "url" => "httpRequest" }
rename => { "query" => "httpRequestQueryParms" }
rename => { "status" => "httpResponseCode" }
rename => { "bytes" => "httpResponseSize" }
rename => { "elapsedTime" => "httpResponseTime" }
rename => { "referer" => "httpReferer" }
rename => { "jsessionId" => "httpJSessionID" }
rename => { "userAgent" => "httpUserAgent" }
convert => { "httpResponseCode" => "integer" }
convert => { "httpResponseSize" => "integer" }
convert => { "httpResponseTime" => "integer" }
}
date {
locale => "en"
match => ["httpTime", "yyyy-MM-dd'T'HH:mm:ss:SSSz"]
timezone => "US/Eastern"
}
}
I'm changing the field name from "datetime" to "httpTime" and then formatting the "httpTime" field.
I'm assuming from the above that my date/time format is wrong on the date match. Can someone please help me format this properly based upon my input format? Thanks!