No log entry after adding date match filter


(Len Rugen) #1

With the date fields below commented out, I get events logged, but with the wrong timestamp. Removing the comment I get nothing logged and no error messages, at least that I can find. The timezone line doesn't matter either.

Any ideal?

Thanks

      grok {
            match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp} %{IPORHOST:s-ip} %{WORD:cs-method} %{URIPATH:cs-uri-stem} %{NOTSPACE:cs-uri-query} %{NUMBER:s-port} %{NOTSPACE:cs-username} %{IPORHOST:clientip} %{NOTSPACE:cs-useragent} %{NOTSPACE:referrer} %{NUMBER:sc-status} %{NUMBER:sc-substatus} %{NUMBER:sc-win32-status} %{NUMBER:time-taken}"]
      }

date {

match => [ "log_timestamp", "YYYY-MM-dd HH:mm:ss" ]

timezone => "US/Central"

}

      geoip { source => "clientip"  target => "geoip"}

      mutate {
          replace => { "[type]" => "iis" }
          lowercase => [ "cs-username" ]
          remove_field => [ "cs-method", "cs-uri-query", "s-port", "referrer", "sc-substatus", "sc-win32-status",
                           "time-taken", "message" ]
      }

(Len Rugen) #2

More research, no solution. With this code in the date block above

      date {
          match => [ "log_timestamp", "YYYY-MM-dd HH:mm:ss" ]
          remove_field => [ "log_timestamp" ]
          target => "log_timestamp_test"
      }

log_timestamp_test has the tiny clock icon and a sample displays as December 21st 2016, 03:11:42.000 and the log_timestamp field is dropped, so I know it's going thru the code :slight_smile:

If I make the target => "@timestamp", I still don't get anything sent to elasticsearch.


(Christian Dahlqvist) #3

It would probably help if you showed an example of an actual event.


(Len Rugen) #4

I tried to add an example last night but I don't see it. Here is a log entry, with some sanitization:

2016-12-10 23:59:58 10.10.10.1 POST /EWS/Exchange.asmx &CorrelationID=;&cafeReqId=f7e635f8-e9ff-4c24-91f9-7259c4d8e5bc; 80 DOMAIN\user 192.168.0.1 Mac+OS+X/10.12.1+(16B2555)+CalendarAgent/384 - 200 0 0 15

I found that adding timezone => "Etc/UTC" fixed the problem.

     date {
          match => [ "log_timestamp", "YYYY-MM-dd HH:mm:ss" ]
          remove_field => [ "log_timestamp" ]
          target => "@timestamp"
          timezone => "Etc/UTC"
      }

I noticed there isn't a "Z" in the timestamp in the this log.

I'm new in ELK, we're just seeing what we can develop.


(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.