# Get the right timestamp for old log files
**URL:** https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943
**Category:** Logstash
**Created:** [July 6, 2015, 11:21am UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943 "2015-07-06T11:21:30Z")
**Posts on this page:** 20
**Page:** 3
### Author:  [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)
#### Post date: [July 8, 2015, 1:37pm UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/41 "2015-07-08T13:37:35Z")
Yes I've been looking at other forums as well like stackoverflow and such and they have the same configuration as you showed. I've tried to take away one row at a time to find a solution but it still gives me the same result. This is not supposed to happen haha.
---
### Author:  [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)
#### Post date: [July 9, 2015, 1:15pm UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/42 "2015-07-09T13:15:13Z")
Okey so I tried doing a new logstash config and I also did a new log file with the same information as the original ones. I managed to get the same result as you when I wrote it out in the console. This makes me wonder, can it be something with Kibana?
---
### Author:  [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)
#### Post date: [July 10, 2015, 8:25am UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/43 "2015-07-10T08:25:22Z")
I'm so confused. I just can't understand why it doesn't take the place as the standard @timestamp. It shows up as a field and everything.
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [July 10, 2015, 8:49am UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/44 "2015-07-10T08:49:26Z")
If you create a minimal example that exhibits the problem we can help you debug it. Start with the example I posted a couple of days ago and add your own filters one by one.
---
### Author:  [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)
#### Post date: [July 10, 2015, 9:08am UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/45 "2015-07-10T09:08:43Z")
I actually copied one basic line from a real log and put it into the example you gave me and tested it out. The "mytimestamp" field actually became the @timestamp field in the console. But in Kibana it doesn't. I think something is up with Kibana.
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [July 10, 2015, 9:45am UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/46 "2015-07-10T09:45:01Z")
Kibana only displays what's in Elasticsearch, and Elasticsearch stores what Logstash gives to it. Focus your attention on what Logstash does. It's unsurprising that the exact example I gave you works. What you should do it "migrate" that configuration towards what you have that doesn't work. At some point things will break and then it shouldn't be too hard to figure out what.
---
### Author:  [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)
#### Post date: [July 10, 2015, 11:30am UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/47 "2015-07-10T11:30:21Z")
So I migrated my config bit by bit. Now I was running the rubydebug in the output. Both the input layer and the filter layer is the exact same as the original logstash config. The only thing that is the difference is the ouput layer because it doesn't connect with elasticsearch.
In the debug I ran the @timestamp field became the same as the "mytimestamp" field that I have created.
---
### Author:  [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)
#### Post date: [July 13, 2015, 7:23am UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/48 "2015-07-13T07:23:30Z")
After some further testing the problem has to do with the output because that's the only difference with my two logstash configs. On the one that does work the output is the rubydebug codec in the console. On the one that doesn't work the output is elasticsearch. I think it has something to do with the indexing that elasticsearch does.
---
### Author:  [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)
#### Post date: [July 14, 2015, 12:59pm UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/49 "2015-07-14T12:59:59Z")
Okey so I've had some days to look into the problem. I used rubydebug to see that the fields were showing the right thing on logstash's part. I used a plugin called HEAD on elasticsearch to see that the fields were fine there. So I think the problem lies in Kibana 4.
---
### Author:  [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)
#### Post date: [July 16, 2015, 3:04pm UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/50 "2015-07-16T15:04:26Z")
After 4 long days I finally solved my problem.
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [July 16, 2015, 4:58pm UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/51 "2015-07-16T16:58:26Z")
And the solution was...?
---
### Author:  [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)
#### Post date: [July 17, 2015, 6:37am UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/52 "2015-07-17T06:37:45Z")
Well let's start with the problem that caused everything. The problem was that I had old docker images lying around so the solution was to simply delete them all.
---
### Author:  [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)
#### Post date: [July 17, 2015, 12:35pm UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/53 "2015-07-17T12:35:58Z")
To be more precise. It is the elasticsearch docker image that is causing the problem. Not the logstash or kibana one.
---
### Author:  [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)
#### Post date: [July 23, 2015, 2:22pm UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/54 "2015-07-23T14:22:23Z")
Okey. After I fixed it I found another problem. Some files were indexed with the right timestamp which was "mytimestamp" and some were indexed with the current timestamp. When I looked this through I found the problem. In my grok filter I have two custom grok filters. Apparently if the first filter works on a event logstash will automatically jump over the next one. So in my case my first grok filter said that it should sort out the paths where "repositories" were included while my next filter made it's own timestamp depending on the time of the file. If the event had a path with the word "repositories" in it the timestamp would become the current timestamp instead of the one written in the log file. Has anyone else experienced the same problem?
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [July 24, 2015, 9:45am UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/55 "2015-07-24T09:45:30Z")
Posting your configuration would help.
---
### Author:  [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)
#### Post date: [July 24, 2015, 9:47am UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/56 "2015-07-24T09:47:34Z")
```
input {
syslog {
port => 5514
codec => "json"
}
file {
path => "/var/externallogs_maven/data"
type => "nexus-log"
start_position => "beginning"
}
}
filter {
grok {
type => "nexus-log"
match => [
"message", "\b\w+\b\s/nexus/content/repositories/(?[^/]+)",
"message", "(?%{MONTHDAY}/%{MONTH}/%{YEAR}:%{HOUR}:%{MINUTE}:%{SECOND} %{ISO8601_TIMEZONE})"
]
}
date{
match => ["mytimestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
}
}
output {
elasticsearch{
host => es
port => 9300
cluster => "elkjepp"
protocol => "transport"
}
stdout { codec => rubydebug }
}
```
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [July 24, 2015, 9:53am UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/57 "2015-07-24T09:53:49Z")
Okay. You have a single grok filter with multiple expressions. With the default true value of [`break_on_match`](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#plugins-filters-grok-break_on_match) grok will stop once it gets a match. You need to disable that or split your multi-expression grok filter into two separate filters.
---
### Author:  [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)
#### Post date: [July 24, 2015, 9:57am UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/58 "2015-07-24T09:57:44Z")
Oh I see. So where do I put the "break\_on\_match" expression in my logstash configuration?
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [July 24, 2015, 10:25am UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/59 "2015-07-24T10:25:23Z")
As evidenced by the documentation I linked to it's a parameter to the grok filter.
---
### Author:  [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)
#### Post date: [July 27, 2015, 9:43am UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943/61 "2015-07-27T09:43:27Z")
I finally managed to get everything to work out just fine. It was me who managed to left an "i" in my config. Very clumsy of me. Although now that I have populated the standard @timestamp is there a way to remove the "mytimestamp" filter that I have created?
[Previous page](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943.md?page=2)
[Next page](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943.md?page=4)