@timestamp not set when parsing nginx logs


#1

For a long time I assumed @timestamp was being parsed correctly; the time in the kibana Time field matches the @timestamp field, which matched the data being logged; however, logstash crashed over the weekend so I started it up this morning, and all the missed logs have @timestamp set to today, completely different to the time in the log.

I'm using the following pattern...

%{IPORHOST:clientip} %{NGUSER:ident} %{NGUSER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer})(?:;|) %{QS:agent}

...which the grok debugger seems to be fine with (some data removed)...

{
"timestamp": [
  [
    "22/Nov/2015:08:00:43 +0100"
  ]
],
"MONTHDAY": [
  [
    "22"
  ]
],
"MONTH": [
  [
    "Nov"
  ]
],
"YEAR": [
  [
    "2015"
  ]
],
"TIME": [
  [
    "08:00:43"
  ]
],
"HOUR": [
  [
    "08"
  ]
],
"MINUTE": [
  [
    "00"
  ]
],
"SECOND": [
  [
    "43"
  ]
],
"INT": [
  [
    "+0100"
  ]
],

...when parsing the following...

x.x.x.x - - [22/Nov/2015:08:00:43 +0100] "GET /secret HTTP/1.1" 304 0 "" "Mozilla/5.0 (Windows NT 6.1; rv:42.0) Gecko/20100101 Firefox/42.0"

(Magnus B├Ąck) #2

The grok filter doesn't populate @timestamp, that's the job of the date filter. Do you have a date filter, and if yes, how is it configured?


#3

That'll be my mistake then. I assumed since I parsed it correctly with grok and I had an @timestamp field in the index, that it was done for me.


(system) #4