Im trying to process logs from liferay (a CMS application), im trying to achieve @timestamp to be same as log event timestamp but the logevent timestamp has only time and miliseconds no date
log file name:
liferay.2018-06-06.log
my log sample
15:27:16,370 INFO [ajp-bio-8009-exec-39][LiferayUtility:185] content funcation
so my query, can we get the date as a field from the file name as my log file-names contain date in it ?, so that i can merge both date field and logtime
it joins the bolded parts into a new field called datetime, where then the date filter is applied.
As for the rubyexception error, I forgot to quote the field names. Corrected that above, should work now.
Also, keep in mind that by default the date filter assumes that the date is based on whatever your machine's local timezone is and converts it to UTC, so you might see a few hours' offset if your machine is not on UTC timezone. If you don't want that to happen, you need to specify a timezone yourself, like mentioned here.
and also Thank you for the explanation of the code.
Q) But in kibana the time is not matching is it due to timezone ? is this the offset you are talking about, what should i do to get the same time zone as the log event and also there is no time zone in the sample logoutput? kindly explain
That is probably the reason, correct. If you want to have it identical to the log you can add a timezone to the date filter, like this
date {
match => [ "datetime", "yyyy-MM-dd HH:mm:ss" ]
timezone => "Etc/UTC"
}
It's not a hard requirement, for the most part there are already filters out there that do what you need to do (with varying degrees of complexity).
But it will pay off to get accustomed to Ruby if you plan to write complex or very specific Logstash configurations, like the one above.
For instance, what you needed could probably be done with a series of existing filter plugins, but having a single Ruby line for that is much cleaner and also more performant.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.