Hi,
I am using logstash 1.5.0 on a windows platform. I am new to logstash. I have written logstash configuration for parsing my log file. The configuration is as follows
input{
file {
path => "C:/dummy.log"
start_position => "beginning"
codec => multiline {
pattern => "^%{DATE}"
negate => true
what => previous
}
}
}
filter {
grok {
match=>[
"message","(?%{DATE}%{SPACE}(?%{HOUR}:%{MINUTE}))-(?%{WORD}-%{LOGLEVEL:loggingLevel}):%{GREEDYDATA}",
"message","(?%{DATE}%{SPACE}(?%{HOUR}:%{MINUTE}))-(?%{WORD}-%{LOGLEVEL:loggingLevel}):%{DATA}Run:%{DATA}%{NUMBER:runNo},%{DATA}ContextKey:%{DATA}%{NUMBER:ctx:integer},%{DATA}COBDate:%{DATA}(?%{MONTHDAY}-%{MONTH}-%{YEAR}),%{DATA}Step:%{GREEDYDATA:step},%{DATA}Type:%{GREEDYDATA:type}",
"message","(?%{DATE}%{SPACE}(?%{HOUR}:%{MINUTE}))-(?%{WORD}-%{LOGLEVEL:loggingLevel}):%{DATA}Run:%{DATA}%{NUMBER:runNo:integer},%{DATA}ContextKey:%{DATA}%{NUMBER:ctx:integer},%{DATA}COBDate:%{DATA}(?%{MONTHDAY}-%{MONTH}-%{YEAR}),%{DATA}Step:%{GREEDYDATA:step},%{DATA}Type:%{GREEDYDATA:type},%{DATA}Status:%{DATA}%{WORD:status}%{GREEDYDATA}",
"message","(?%{DATE}%{SPACE}(?%{HOUR}:%{MINUTE}))-(?%{WORD}-%{LOGLEVEL:loggingLevel}):%{DATA}Run:%{DATA}%{NUMBER:runNo},%{DATA}ContextKey:%{DATA}%{NUMBER:ctx:integer},%{DATA}COBDate:%{DATA}(?%{MONTHDAY}-%{MONTH}-%{YEAR}),%{DATA}Step:%{GREEDYDATA:step},%{DATA}Type:%{GREEDYDATA:type},%{DATA}Time:%{DATA}%{NUMBER:timeTaken:integer}%{GREEDYDATA}"
]
}
date {
match => ["loggedtime","MM/dd/yyyy HH:mm"]
}
}
output {
if ("_grokparsefailure" in [tags]) {
file {
path => "C:/grok-error.log"
}
}else {
file {
path => "C:/matchedlogs.log"
}
}
}
I have a sample log file with the following entries
06/16/15 11:30-X-INFO: Run:1,ContextKey: 1, COBDate: 31-May-2015, Step:A, Type: Whole
06/16/15 11:40-X-INFO: Run:1, ContextKey: 1, COBDate: 31-May-2015, Step: A, Type: Individual, Status: Running
06/16/15 11:45-X-INFO: Run:1, ContextKey: 1, COBDate: 31-May-2015, Step: B, Type: Individual, Status: Completed
06/16/15 11:45-X-INFO: Run:1, ContextKey: 1, COBDate: 31-May-2015, Step: B, Type: Individual, Time: 300
06/16/15 11:46-X-INFO: Run:1, ContextKey: 1, COBDate: 31-May-2015, Step: C, Type: Individual, Status: Running
06/16/15 11:51-X-INFO: Run:1, ContextKey: 1, COBDate: 31-May-2015, Step: C, Type: Individual, Status: Completed
06/16/15 11:51-X-INFO: Run:1, ContextKey: 1, COBDate: 31-May-2015, Step: C, Type: Individual, Time: 300
06/16/15 11:55-X-INFO: Run:1,ContextKey: 1, COBDate: 31-May-2015, Step: D, Type: Whole
06/16/15 11:55-X-INFO: Run:1, ContextKey: 1, COBDate: 31-May-2015, Step: D, Type: Whole, Time: 900
06/16/15 12:30-X-INFO: Run:2,ContextKey: 1, COBDate: 31-May-2015, Step: A, Type: Whole
06/16/15 12:40-X-INFO: Run:2, ContextKey: 1, COBDate: 31-May-2015, Step: B, Type: Individual, Status: Running
06/16/15 12:44-X-INFO: Run:2, ContextKey: 1, COBDate: 31-May-2015, Step: B, Type: Individual, Status: Completed
06/16/15 12:44-X-INFO: Run:2, ContextKey: 1, COBDate: 31-May-2015, Step: B, Type: Individual, Time: 240
06/16/15 12:46-X-INFO: Run:2, ContextKey: 1, COBDate: 31-May-2015, Step: C, Type: Individual, Status: Running
06/16/15 12:55-X-INFO: Run:2, ContextKey: 1, COBDate: 31-May-2015, Step: C, Type: Individual, Status: Completed
06/16/15 12:55-X-INFO: Run:2, ContextKey: 1, COBDate: 31-May-2015, Step: C, Type: Individual, Time: 540
06/16/15 12:58-X-INFO: Run:2,ContextKey: 1, COBDate: 31-May-2015, Step: D, Type: Whole
06/16/15 12:58-X-INFO: Run:2, ContextKey: 1, COBDate: 31-May-2015, Step: A, Type: Whole, Time: 1680
I have individual lines and the grok patterns at https://grokdebug.herokuapp.com/ . i have also run individual logs with stdin as input. The grok filter seems to be working fine when run individually. However,when i run all the above logs in a file and run logstash, i noticed that lines from line no 13 are getting skipped. They are neither logged as grokparsefailure nor in the logs (as in the configuration above).
Request your help on finding out what the issue is.
Thanks and Regards,
Chaitanya Varanasi