Grok filter to ignore few lines of the logs and parse unstructured log file


(rakesh) #1

Hi,

I am new to ELK, saying that, i m struggling with grok filters on parsing a robotic logs (generated by robots)
The logs are unstructured with no certain pattern to write a grok filter.

below are the samples of the log:

KTLogging Service
KLATencor.Infrastructure.Logging.Listeners.KTTextWriterListener, Version=7.1.1.23303, Culture=neutral, PublicKeyToken=null
Log started at: 2018-06-12 15:28:22.531
ASD SW Version::
Event Type Timestamp Log Message Category User Thread ID Error Code Application Machine
2018-06-12 15:28:27.072 KT Common Platform Service has started. KTCPInfrastructure NT AUTHORITY\SYSTEM 4 0x0 KTCommonPlatformService FEC
2018-06-12 15:28:27.067 Error in cleaning up remoting temp directory.Could not find a part of the path 'C:\temp\KTCP_remoting_config'. KTCPInfrastructure NT AUTHORITY\SYSTEM 4 0x0 KTCPInfrastructure.Utilities FEC
2018-06-12 15:28:27.078 Waiting for EDS to publish its server objects.KTCP will wait for the configured time:10000 milliseconds KTCPInfrastructure NT AUTHORITY\SYSTEM 4 0x0 KTCommonPlatformService FEC
2018-06-12 15:28:27.079 EDS has signaled that it has published the server objects OR the wait time for EDS to be ready is over. Proceeding to publish KTCP server objects. Some server objects will not be valid if EDS is not ready yet. KTCPInfrastructure NT AUTHORITY\SYSTEM 4 0x0 KTCommonPlatformService FEC
2018-06-12 15:28:27.082 KTCP Application 'KTCS' does not have an alias name. KTCPInfrastructure NT AUTHORITY\SYSTEM 9 0x0 KTCommonPlatformService FEC
2018-06-12 15:28:28.432 Registered the MasterProcessController for KTCP application KTCS successfully. KTCPInfrastructure NT AUTHORITY\SYSTEM 9 0x0 KTCommonPlatformService FEC
2018-06-12 15:28:28.436 Registered the ComponentRegistrar for KTCP application KTCS successfully. KTCPInfrastructure NT AUTHORITY\SYSTEM 9 0x0 KTCommonPlatformService FEC
2018-06-12 15:28:28.441 Registered the SessionRegistrar for KTCP application KTCS successfully. KTCPInfrastructure NT AUTHORITY\SYSTEM 9 0x0 KTCommonPlatformService FEC
2018-06-12 15:28:28.441 All KTCP server objects are registered and the KTCP service is ready to receive requests. KTCPInfrastructure NT AUTHORITY\SYSTEM 4 0x0 KTCommonPlatformService FEC
2018-06-12 15:28:29.278 Error in cleaning up remoting temp directory.Could not find a part of the path 'C:\temp\KTCP_remoting_config'. KTCPInfrastructure NT AUTHORITY\SYSTEM 4 0x0 KTCPInfrastructure.Utilities FEC
2018-06-12 15:28:29.289 KT Common Platform Service has stopped. KTCPInfrastructure NT AUTHORITY\SYSTEM 4 0x0 KTCommonPlatformService FEC

I don't know how to write a grok filter to parse this.

Below is the config file i have created:

input {
stdin{

}

}

filter {
grok {
match => {"message" => "<%{WORD:loglevel}>%{TIMESTAMP_ISO8601:timestamp}%{GREEDYDATA:logmessage}'%{PATH:path}'%{WORD:category}%{WORD:user}%{NUMBER:threadid}%{BASE16NUM:errorcode}%{WORD:application}%{WORD:machine}"}
}
}
output {
file {
path => "D:\logstash-6.3.0\data1\output8.csv"
}
}

but it gives irrelevant results.


(rakesh) #2

any help would be appreciated.

Event Type Timestamp Log Message Category User Thread ID Error Code Application Machine
2018-06-12 15:28:27.072 KT Common Platform Service has started. KTCPInfrastructure NT AUTHORITY\SYSTEM 4 0x0 KTCommonPlatformService FEC

(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.