Issue with logstash configuration for log file having mix of multiline as well as single level events

Elastic Stack Logstash
1

Hi ,
I am newbie to logstash and i am using logstash 1.5.0. I have a log file (which is a historical log file ,as of now). I have written the following configuration. however i am unable to get any values. I have used the Grok Debugger to test the data and there were matches.

Update: I notice that whenever the file is changed, the filtering is activated and reading the file from beginning. However, if the file is historical, no new changes would be made. How do we make Logstash to read the entire file from beginning (i have specified the start_position to beginning.

Log File format :

04/15/15 18:40-XYZ-WARNING: Some Data
04/15/15 18:40-XYZ-WARNING: Some Data
04/15/15 18:40-XYZ-WARNING: Some Data
More Data on a new line. This line is a part of the previous line

I have the following config

input{
file {
path => "C:/a.log"
type => "syslog"
start_position => "beginning"
sincedb_path=>null
codec => multiline {
pattern => "^%{DATE}"
negate => true
what => previous
}
}
}

filter {
grok {
match => { "message" => "%{DATE}%{SPACE}(?%{HOUR}:%{MINUTE})-(?%{WORD}-%{LOGLEVEL}):%{GREEDYDATA}" }
}
}

output {
stdout {}
}

However i am unable to get any output matching. Request your help in letting me know where i am getting things wrong. Any pointers would be helpful

Thanks and Regards,
Chaitanya V

2

This is a sincedb issue, if Logstash has seen the file at any point in time it will never go back the beginning of the file despite start_position being set.
Delete the since.db file and you should be good.

If you are testing this file continually it's best to use stdin and then pipe the file into LS.

3

Thanks a lot for the prompt reply @warkolm . I have deleted the sincedb file on the my login user folder and things started working well.