HI Team,
I have installed ELK 6.4.2. I have Pentaho logs generated from my applications. I have successfully installed and configured Elastic, Kibana, Logstash and Filebeat. The generated logs are consumed by the Filebeat and I am able to index them and see the the data in the discover tab of Kibana.
I am trying to index the logs generated by the ETL tool called Pentaho DI. I am using Grok pattern in the Logstash config file. here are a few lines of the log messages that are generated by the Pentaho ETL tool.
2019/02/04 15:19:42 - Pick JOB_NAME.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2019/02/04 15:19:42 - Get JOB table.0 - Finished processing (I=1, O=0, R=1, W=1, U=0, E=0)
2019/02/04 15:19:42 - Does JOB_NAME Exist?.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2019/02/04 15:19:42 - Get Inprocess Dependent jobs.0 - Finished processing (I=1, O=0, R=1, W=1, U=0, E=0)
2019/02/04 15:19:42 - Filter rows.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2019/02/04 15:19:42 - Set up 'In Process'.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
I am able to index the files via filebeat and see the data in the Discover tab. However, I am having issues in querying the data.
I have set the below pattern for the logs in the grok filter
grok {
match => {"message" => "%{DATESTAMP:LogTimeStamp} - %{GREEDYDATA:StepName}- %{GREEDYDATA:LogMessage}" }
}
I am able to see the 3 fileds LogTimeStamp, StepName and LogMessage. The search query returns values only when the complete text in the field values are used. I need to be able to search partial words in the fields.
for eg: the below query does not return any result. however when LogMessage is substituted for the whole text, "Finished processing (I=1, O=0, R=1, W=1, U=0, E=0)", then it does result in showing up the data.
GET filebeat-6.4.2-2019.01.23/_search
{
"query" : {
"term" : {
"LogMessage": "R=1"
}
}
}
Please suggest me how the GROK pattern should be used to parse the above logs so that I can query the fields just by searching for lines of logs which has E=1 or R=1 etc.
Also, Once I am able to filter by the search string. I would like to see the next 5 to 10 lines in the logs to access the error based on the log messages.
Thanks a lot in advance