Parsing Pentaho logs using Grok in Logstash config

HI Team,

I have installed ELK 6.4.2. I have Pentaho logs generated from my applications. I have successfully installed and configured Elastic, Kibana, Logstash and Filebeat. The generated logs are consumed by the Filebeat and I am able to index them and see the the data in the discover tab of Kibana.

I am trying to index the logs generated by the ETL tool called Pentaho DI. I am using Grok pattern in the Logstash config file. here are a few lines of the log messages that are generated by the Pentaho ETL tool.

2019/02/04 15:19:42 - Pick JOB_NAME.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)

2019/02/04 15:19:42 - Get JOB table.0 - Finished processing (I=1, O=0, R=1, W=1, U=0, E=0)

2019/02/04 15:19:42 - Does JOB_NAME Exist?.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)

2019/02/04 15:19:42 - Get Inprocess Dependent jobs.0 - Finished processing (I=1, O=0, R=1, W=1, U=0, E=0)

2019/02/04 15:19:42 - Filter rows.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)

2019/02/04 15:19:42 - Set up 'In Process'.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)

I am able to index the files via filebeat and see the data in the Discover tab. However, I am having issues in querying the data.

I have set the below pattern for the logs in the grok filter

grok {

match => {"message" => "%{DATESTAMP:LogTimeStamp} - %{GREEDYDATA:StepName}- %{GREEDYDATA:LogMessage}" }

}

I am able to see the 3 fileds LogTimeStamp, StepName and LogMessage. The search query returns values only when the complete text in the field values are used. I need to be able to search partial words in the fields.

for eg: the below query does not return any result. however when LogMessage is substituted for the whole text, "Finished processing (I=1, O=0, R=1, W=1, U=0, E=0)", then it does result in showing up the data.

GET filebeat-6.4.2-2019.01.23/_search

{

"query" : {

"term" : {

  "LogMessage": "R=1"

}

}

}

Please suggest me how the GROK pattern should be used to parse the above logs so that I can query the fields just by searching for lines of logs which has E=1 or R=1 etc.

Also, Once I am able to filter by the search string. I would like to see the next 5 to 10 lines in the logs to access the error based on the log messages.

Thanks a lot in advance

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.