# Get the right timestamp for old log files
**URL:** https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943
**Category:** Logstash
**Created:** [July 6, 2015, 11:21am UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943 "2015-07-06T11:21:30Z")
**Posts on this page:** 20
**Page:** 2
### Author:  [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)
#### Post date: [July 7, 2015, 1:21pm UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/21 "2015-07-07T13:21:29Z")
So I just basically write target =\> "@timestamp" ? Or do I just skip the "@timestamp" part?
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [July 7, 2015, 1:27pm UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/22 "2015-07-07T13:27:45Z")
You don't have to set `target` since it defaults to @timestamp.
---
### Author:  [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)
#### Post date: [July 7, 2015, 1:31pm UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/23 "2015-07-07T13:31:39Z")
So the date filter makes my "mytimestamp" field go to the original "@timestamp" field by default? The reaopn I'm curious is because it didn't really do it by default for me. Maybe I wrote something wrong in my date filter?
```
date {
match => ["mytimestamp", "dd/MMM/YYYY:HH:mm:ss +SSSS"]
}
```
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [July 7, 2015, 1:36pm UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/24 "2015-07-07T13:36:21Z")
> So the date filter makes my "mytimestamp" field go to the original "@timestamp" field by default?
Yes. That's what I said.
> The reaopn I'm curious is because it didn't really do it by default for me. Maybe I wrote something wrong in my date filter?
Possibly yes. For example, "+SSSS" doesn't look right. Are you sure you don't mean "ZZZ", for parsing a timezone offset? The date filter will cry out in the logs if it's not able to parse a timestamp (as you saw when you still had `locale => "sv"` in place).
---
### Author:  [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)
#### Post date: [July 7, 2015, 1:45pm UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/25 "2015-07-07T13:45:55Z")
The reason that the "+SSS" is there is because I'm trying to parse "15:42:51.541" haha! Don't know if there is a pattern for millieseconds.
EDIT: Sorry that was all wrong. I'm trying to parse this =\> 22/Jun/2015:02:37:17 +0000.
I actually don't know what the last zeros stand for.
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [July 7, 2015, 2:30pm UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/26 "2015-07-07T14:30:52Z")
That's the timezone offset, which ZZZ should be able to parse.
---
### Author:  [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)
#### Post date: [July 7, 2015, 2:35pm UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/27 "2015-07-07T14:35:25Z")
Should it be something like this?
```
match => ["mytimestamp", "dd/MMM/YYYY:HH:mm:ss +ZZZZ"]
```
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [July 8, 2015, 6:37am UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/28 "2015-07-08T06:37:21Z")
The plus sign is part of the offset and will be parsed. As I said "ZZZ" should be fine. "ZZZZ" might also work. I'm not sure if there's any difference between them. The Joda-Time documentation isn't amazingly clear on this.
---
### Author:  [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)
#### Post date: [July 8, 2015, 7:25am UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/29 "2015-07-08T07:25:30Z")
Hi I've tried them both and unfortunately none of them seems to work. I keep getting the message
:message=\>"Failed parsing date from field", :field=\>"mytimestamp", :value=\>"22/Jun/2015:23:59:51", :exception=\>java.lang.IllegalArgumentException: Invalid format: "22/Jun/2015:23:59:51" is too short, :level=\>:warn}
This states that the format is to short so my best guess is that the match under the date filter doesn't match the custom grok filter.
Here is what it looks like.
```
grok {
type => "nexus-log"
patterns_dir => "./config-dir/patterns"
match => [
"message", "\b\w+\b\s/nexus/content/repositories/(?[^/]+)",
"message", "(?%{MONTHDAY}/%{MONTH}/%{YEAR}:%{HOUR}:%{MINUTE}:%{SECOND})"
]
}
date{
match => ["mytimestamp", "dd/MMM/YYYY:HH:mm:ss +ZZZZ"]
}
```
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [July 8, 2015, 7:27am UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/30 "2015-07-08T07:27:47Z")
Yes, the grok expression doesn't include the timezone offset in the `mytimestamp` field.
---
### Author:  [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)
#### Post date: [July 8, 2015, 7:31am UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/31 "2015-07-08T07:31:14Z")
Is there a ceartain format for the timezone offset that I can include in my custom filter?
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [July 8, 2015, 7:39am UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/32 "2015-07-08T07:39:02Z")
ISO8601\_TIMEZONE should work.
---
### Author:  [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)
#### Post date: [July 8, 2015, 7:44am UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/33 "2015-07-08T07:44:53Z")
I tried it and got this error message.
:message=\>"Failed parsing date from field", :field=\>"mytimestamp", :value=\>"22/Jun/2015:00:02:32 +0000", :exception=\>java.lang.IllegalArgumentException: Invalid format: "22/Jun/2015:00:02:32 +0000" is malformed at "0000", :level=\>:warn}
This is how the config looks like right now
```
grok {
type => "nexus-log"
patterns_dir => "./config-dir/patterns"
match => [
"message", "\b\w+\b\s/nexus/content/repositories/(?[^/]+)",
"message", "(?%{MONTHDAY}/%{MONTH}/%{YEAR}:%{HOUR}:%{MINUTE}:%{SECOND} +%{ISO8601_TIMEZONE})"
]
}
date{
match => ["mytimestamp", "dd/MMM/YYYY:HH:mm:ss +ZZZZ"]
}
```
What does "malformed" mean?
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [July 8, 2015, 7:58am UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/34 "2015-07-08T07:58:58Z")
It seems ZZZ and ZZZZ weren't acceptable after all but Z and ZZ works.
And again, the plus sign is part of the timezone offset, both the one recognized by ISO8601\_TIMEZONE and the one parsed by Z or ZZ. Don't include it in either place. I've verified that the following filters work with the example input you've provided:
```
filter {
grok {
match => [
"message",
"(?%{MONTHDAY}/%{MONTH}/%{YEAR}:%{HOUR}:%{MINUTE}:%{SECOND} %{ISO8601_TIMEZONE})"
]
}
date {
match => ["mytimestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
}
}
```
---
### Author:  [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)
#### Post date: [July 8, 2015, 8:04am UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/35 "2015-07-08T08:04:00Z")
Yes it works. Both Logstash, Elasticsearch and Kibana start without any error messages. Although the standard "@timestamp" still says today's current date. I'm gonna check everything out so it checks out as it's supposed to. Very strange.
---
### Author:  [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)
#### Post date: [July 8, 2015, 9:23am UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/36 "2015-07-08T09:23:05Z")
Okey so I have been toying around a little bit with the logstash configuration. Apparently when I skip the "+" sign in front of "Z" I don't get any error messages although Kibana 4 doesn't recognize my custom filter "mytimestamp". When I have the "+" sign in front of "Z" it recognizes my custom filter but I keep getting error messages saying that it has a invalid format.
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [July 8, 2015, 10:20am UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/37 "2015-07-08T10:20:45Z")
Use the `@timestamp` field in Kibana and make sure it's correctly populated with the timestamp from your log. Do not try to get Kibana to use `mytimestamp`.
---
### Author:  [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)
#### Post date: [July 8, 2015, 10:53am UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/38 "2015-07-08T10:53:56Z")
That is exactly the problem I'm having. The standard @timestamp is populated with the timestamp that is generated with the time I'm indexing the log instead of being populated with the timestamp from the events in the log itself. How do I make sure it's populated with the time from the log itself? I thought you had to use the date filter and in some way make Kibana use my own timestamp as the standard one.
---
### Author:  [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)
#### Post date: [July 8, 2015, 11:35am UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/39 "2015-07-08T11:35:35Z")
Gonna try the "mutate" filter.
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [July 8, 2015, 12:29pm UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/40 "2015-07-08T12:29:20Z")
With the configuration you've shown the `@timestamp` field should be populated with the date from your log entries and everything should be fine. If you can produce a minimal example that exhibits the problem it'll be possible to help out.
The following example shows what I did to test the grok and date filter combination, and it works. So you're obviously doing _something_ differently that causes things to break. I suggest you start from this example and step by step turn it into what you actually have and observe when it breaks.
```
$ cat data
22/Jun/2015:00:02:32 +00:00
$ cat test.config
input { stdin { codec => plain } }
output { stdout { codec => rubydebug } }
filter {
grok {
match => [
"message",
"(?%{MONTHDAY}/%{MONTH}/%{YEAR}:%{HOUR}:%{MINUTE}:%{SECOND} %{ISO8601_TIMEZONE})"
]
}
date {
match => ["mytimestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
}
}
$ /opt/logstash/bin/logstash -f test.config < data
{
"message" => "22/Jun/2015:00:02:32 +00:00",
"@version" => "1",
"@timestamp" => "2015-06-22T00:02:32.000Z",
"host" => "redacted",
"mytimestamp" => "22/Jun/2015:00:02:32 +00:00"
}
```
[Previous page](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943.md?page=1)
[Next page](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943.md?page=3)