Yes I've been looking at other forums as well like stackoverflow and such and they have the same configuration as you showed. I've tried to take away one row at a time to find a solution but it still gives me the same result. This is not supposed to happen haha.
Okey so I tried doing a new logstash config and I also did a new log file with the same information as the original ones. I managed to get the same result as you when I wrote it out in the console. This makes me wonder, can it be something with Kibana?
I'm so confused. I just can't understand why it doesn't take the place as the standard @timestamp. It shows up as a field and everything.
If you create a minimal example that exhibits the problem we can help you debug it. Start with the example I posted a couple of days ago and add your own filters one by one.
I actually copied one basic line from a real log and put it into the example you gave me and tested it out. The "mytimestamp" field actually became the @timestamp field in the console. But in Kibana it doesn't. I think something is up with Kibana.
Kibana only displays what's in Elasticsearch, and Elasticsearch stores what Logstash gives to it. Focus your attention on what Logstash does. It's unsurprising that the exact example I gave you works. What you should do it "migrate" that configuration towards what you have that doesn't work. At some point things will break and then it shouldn't be too hard to figure out what.
So I migrated my config bit by bit. Now I was running the rubydebug in the output. Both the input layer and the filter layer is the exact same as the original logstash config. The only thing that is the difference is the ouput layer because it doesn't connect with elasticsearch.
In the debug I ran the @timestamp field became the same as the "mytimestamp" field that I have created.
After some further testing the problem has to do with the output because that's the only difference with my two logstash configs. On the one that does work the output is the rubydebug codec in the console. On the one that doesn't work the output is elasticsearch. I think it has something to do with the indexing that elasticsearch does.
Okey so I've had some days to look into the problem. I used rubydebug to see that the fields were showing the right thing on logstash's part. I used a plugin called HEAD on elasticsearch to see that the fields were fine there. So I think the problem lies in Kibana 4.
After 4 long days I finally solved my problem.
And the solution was...?
Well let's start with the problem that caused everything. The problem was that I had old docker images lying around so the solution was to simply delete them all.
To be more precise. It is the elasticsearch docker image that is causing the problem. Not the logstash or kibana one.
Okey. After I fixed it I found another problem. Some files were indexed with the right timestamp which was "mytimestamp" and some were indexed with the current timestamp. When I looked this through I found the problem. In my grok filter I have two custom grok filters. Apparently if the first filter works on a event logstash will automatically jump over the next one. So in my case my first grok filter said that it should sort out the paths where "repositories" were included while my next filter made it's own timestamp depending on the time of the file. If the event had a path with the word "repositories" in it the timestamp would become the current timestamp instead of the one written in the log file. Has anyone else experienced the same problem?
Posting your configuration would help.
input {
syslog {
port => 5514
codec => "json"
}
file {
path => "/var/externallogs_maven/data"
type => "nexus-log"
start_position => "beginning"
}
}
filter {
grok {
type => "nexus-log"
match => [
"message", "\b\w+\b\s/nexus/content/repositories/(?<repositories>[^/]+)",
"message", "(?<mytimestamp>%{MONTHDAY}/%{MONTH}/%{YEAR}:%{HOUR}:%{MINUTE}:%{SECOND} %{ISO8601_TIMEZONE})"
]
}
date{
match => ["mytimestamp", "dd/MMM/YYYY:HH:mm:ss Z" ]
}
}
output {
elasticsearch{
host => es
port => 9300
cluster => "elkjepp"
protocol => "transport"
}
stdout { codec => rubydebug }
}Okay. You have a single grok filter with multiple expressions. With the default true value of break_on_match grok will stop once it gets a match. You need to disable that or split your multi-expression grok filter into two separate filters.
Oh I see. So where do I put the "break_on_match" expression in my logstash configuration?
As evidenced by the documentation I linked to it's a parameter to the grok filter.
I finally managed to get everything to work out just fine. It was me who managed to left an "i" in my config. Very clumsy of me. Although now that I have populated the standard @timestamp is there a way to remove the "mytimestamp" filter that I have created?