Converting time into @timestamp

Elastic Stack Logstash
1

I'm trying to get my time into @timestamp but are failing.

Lines looks like this "20151012 08:57:43 ..."

And I use the following config:

grok {
match => { "message" => "(?\d{1,8}.\d{1,2}[:]?\d{1,2}[:]\d{1,2})%{SPACE}% ... }

  }

date {
locale => "en"
match => ["date", "YYYYMMdd HH:mm:ss"]
}

I get no errors.
Do I need to convert "date"?

2

Your date filter works fine:

$ cat test.config 
input { stdin {} }
output { stdout { codec => rubydebug } }
filter {
  date {
    match => ["message", "YYYYMMdd HH:mm:ss"]
  }
}
$ echo '20151012 08:57:43' | /opt/logstash/bin/logstash -f test.config
Logstash startup completed
{
       "message" => "20151012 08:57:43",
      "@version" => "1",
    "@timestamp" => "2015-10-12T06:57:43.000Z",
          "host" => "lnxolofon"
}
Logstash shutdown completed

Have you verified that the date field contains what you expect? Have you tried cranking up logging with --verbose or --debug?

3

Yes "date" field works. I can see that in Kibana.

So then I assume it's something I missed in the Kibana config?
Should I start a new thread in that area?

4

Sorry, I don't get what the problem is. First you said that @timestamp isn't populated but now you say that the date filter works. To me that's contradictory.

5

It's not.
The "date" field has the correct information, i.e. 20151012 08:57:43.
But @timestamp is not populated correctly with that information when I see it Kibana, instead @timestamp has the date and time of when I imported the data.

6

Sorry, I misread. Okay, so date is okay and my previous example indicated that the date filter works with the input given and produces a @timestamp with the expected result. I suggest you try the same thing on your side, preferably with a stdout { codec => rubydebug } output. If that also works, inspect the resulting document directly in Elasticsearch. Somewhere along this chain you're going to find the problem.

7

No worries!
I tried that and it gave me some strange output

"message" => "20151004 18:38:23 ...",
"@timestamp" => "2015-01-04T17:38:23.000Z",
"date" => "20151004 18:38:23",

How is that possible?

8

Your date pattern is wrong. I suspect you have "mm" instead of "MM" for the month. The one-hour difference in the time is because @timestamp is UTC and your timezone is UTC+1 (on Jan 4 anyway; on Oct 4 it's probably UTC+2).

9

Actually I had DD instead of dd.

Now it's all working.

Thanks!