Is it possible to get Date from filename as field

[ajb (Jay)]

HI Elastic team,

Im trying to process logs from liferay (a CMS application), im trying to achieve @timestamp to be same as log event timestamp but the logevent timestamp has only time and miliseconds no date

log file name:
liferay.2018-06-06.log

my log sample
15:27:16,370 INFO [ajp-bio-8009-exec-39][LiferayUtility:185] content funcation

so my query, can we get the date as a field from the file name as my log file-names contain date in it ?, so that i can merge both date field and logtime

my pipeline config
input {

       file {
             path => "/home/foss/liferaylogs/*.log"
             start_position => "beginning"
             sincedb_path   => "/dev/null"      
            }
    }

    filter {
        grok {
             match => { "message" => "%{TIME:logTime}\s*%{LOGLEVEL:logLevel}\s*\[%{GREEDYDATA:logThread}\]\s*\[%{GREEDYDATA:logClassName}:%{NUMBER:logLineNumber}\]\s*%{GREEDYDATA:logMessage}" }
         }

        date {
    	match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
    	remove_field => [ "timestamp" ]
       }
    }

    output {
            stdout {
                codec => rubydebug
            }
    }

my stdout rubydebug

[paz]

It is with some custom Ruby code, e.g.

filter {
    grok {
        match => { "message" => "%{TIME:logTime}\s*%{LOGLEVEL:logLevel}\s*\[%{GREEDYDATA:logThread}\]\s*\[%{GREEDYDATA:logClassName}:%{NUMBER:logLineNumber}\]\s*%{GREEDYDATA:logMessage}" }
    }
    ruby {
        code => "
            event.set('datetime', [event.get('path').split('/')[-1].split('.')[-2], event.get('logTime').split(',')[0]].join(' '))
        "
    }
    date {
    	match => [ "datetime", "yyyy-MM-dd HH:mm:ss" ]
    }
}

Date pattern might need some adjustments though, haven't tested that (also needs a bit of change if you want to keep what I presume is milliseconds.

[ajb (Jay)]

@paz

i

I did not understand what ruby code is trying to do, getting rubyexception error

Below is the logstash log

[paz]

All it supposedly does is get the date part of the file and join it with the time field taken from the grok pattern. So if you have :

path => "/home/foss/liferaylogs/liferay.2018-06-06.log",
logTime => "15:27:16,389"

it joins the bolded parts into a new field called datetime, where then the date filter is applied.
As for the rubyexception error, I forgot to quote the field names. Corrected that above, should work now.

Also, keep in mind that by default the date filter assumes that the date is based on whatever your machine's local timezone is and converts it to UTC, so you might see a few hours' offset if your machine is not on UTC timezone. If you don't want that to happen, you need to specify a timezone yourself, like mentioned here.

[ajb (Jay)]

yes paz, the code is working now. Thank you paz

and also Thank you for the explanation of the code.

Q) But in kibana the time is not matching is it due to timezone ? is this the offset you are talking about, what should i do to get the same time zone as the log event and also there is no time zone in the sample logoutput? kindly explain

stdout rubybebug

Q) In future do i have to learn ruby to get better at writing config scripts in logstash

[paz]

@ajb

Q) But in kibana the time is not matching is it due to timezone ? is this the offset you are talking about, what should i do to get the same time zone as the log event and also there is no time zone in the sample logoutput? kindly explain

That is probably the reason, correct. If you want to have it identical to the log you can add a timezone to the date filter, like this

    date {
    	match => [ "datetime", "yyyy-MM-dd HH:mm:ss" ]
        timezone => "Etc/UTC"
    }

@ajb

Q) In future do i have to learn ruby to get better at writing config scripts in logstash

It's not a hard requirement, for the most part there are already filters out there that do what you need to do (with varying degrees of complexity).
But it will pay off to get accustomed to Ruby if you plan to write complex or very specific Logstash configurations, like the one above.

For instance, what you needed could probably be done with a series of existing filter plugins, but having a single Ruby line for that is much cleaner and also more performant.