i want my log event timestamp to replace @timestamp of logstash , what should i do?
{
"message" => "DEBUG",
"@version" => "1",
"@timestamp" => "2015-10-21T07:00:59.979Z", ###this timestamp is of logstash
"host" => "HFX2WS1",
"path" => "C:\Users\egupanm\csv\logs.log",
"timestamp" => "2015-10-21 12:30:59",# this is my timestamp
"log_level" => "TRACE",
"line" => 16,
"time" => 45059,
"difference" => 26
}
How can i replace @timestamp with my timestamp??
Please do not give me link of other thread because i have seen them did exactly what is written still i am not able to achieve my usecase. So please help .
Use the date filter (as I assume other threads have suggested). If you didn't get that to work show us what you've tried so far and why that wasn't satisfactory.
Through this config i am trying to add those event in which time gap is more than 20 seconds and corresponding log level .
Now i want to replace logstash @timestamp with my timestamp field so that i can see that in kibana.
What should i possibly add? because i had to break my timestamp in order to get difference so date filter pattern is not matching so please tell me the pattern ..
my timestamp looks like this: 2015-10-21 12:31:56
var1 and var2 are variables that i added in ruby filter file ruby.rb. I did this inorder to fulfil my use case and i was able to achieve it. but now i want to replace timestamp .
Again, use the date filter. Finding an example of how to use it to parse ISO8601 dates like the one in your timestamp field should be easy. You may be able to use the "ISO8601" date format pattern instead of a "YYYY-..." style pattern.
Thank you but it got resolved there was some error regarding pattern but i am come up with new issues
the timestamp shown in logstash and kibana differs
how is this possible?? when logstash gives it timestamp "@timestamp" => "2015-10-21T16:09:54.077Z"
and kiban gives @timestamp:October 21st 2015, 21:41:00.864
How do you know it's the same message? If you suspect Kibana might be doing something weird keep in mind that you can always fetch a document directly from ES.
Be systematic and simplify your pipeline. Ignore ES for now. Just use the stdout output. Remove the ruby filter. Process a single message from the file. Do you get what you expect? Yes? Continue adding one thing at a time until you get something unexpected. Over and out.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.