Logstash : Obtain metrics from log file based on alert/error message


(Kavin Mjr) #1

We are trying to read a input log file. Assume that the log file is from an application and it might have a combination both alerts and errors. We are trying to separate the alert and error messages and compute metrics on the same.

Senario 1: If there is a space issue on the application and the space error log message appears twice in the log file. I m trying to obtain the following.

  • timeStamp for the alert/error.

  • Server name from the message

  • Metric name eg: Space issue on the server

  • Compute the count.

  • We are able to get the alert/error out of the log and the 1st three fields.

  • We are having trouble computing the count using metrics and combining this with the 1st 3 fields.

Output should look like,

"2017-01-17 22:15:43","RTS HS1","Cannot create log file in the specified location",2

Input File:

2017-01-17 22:11:42,846 INFO [Domain Monitor] [DOM_10142] The update and timeout properties for the master gateway node is set in the domain configuration repository. The row update interval time is set to [8000] milliseconds and the out-of-date row timeout is set to [32000] milliseconds.
2017-01-17 22:12:42,007 INFO [Domain Monitor] [RR_4035] SQL Alert [ORA-00911] INFA RTS Job5, RTS DB2- There is a problem connecting to your SQL database. This could be a temporary SQL alert network issue, or the schema changed in one of your tables.
2017-01-17 22:12:45,007 INFO [Domain Monitor] [RR_4035] SQL Alert [ORA-00911] INFA RTS Job5, RTS DB2- There is a problem connecting to your SQL database. This could be a temporary SQL alert network issue, or the schema changed in one of your tables.
2017-01-17 22:14:43,060 INFO [Domain Monitor] [DOM_10155] No master gateway found in the domain configuration repository. Setting this node to master gateway node.
2017-01-17 22:15:43,060 INFO [Domain Monitor] SPACE Alert [LM_2006] INFA RTS Job1, RTS HS1- Unable to create log file [/base path/WorkflowLogs/wf_11245.log.bin]. Cannot create log file in the specified location
2017-01-17 22:16:43,062 INFO [Domain Monitor] [DOM_10152] Updated data row for node [node01_host1].
2017-01-17 22:17:43,060 INFO [Domain Monitor] SPACE Alert [LM_2006] INFA RTS Job1, RTS HS1- Unable to create log file [/base path/WorkflowLogs/wf_11245.log.bin]. Cannot create log file in the specified location
2017-01-17 22:18:43,062 INFO [Domain Monitor] [DOM_10152] Updated data row for node [node01_host1].
2017-01-17 22:19:43,060 INFO [Domain Monitor] SPACE Alert [LM_2006] INFA RTS Job1, RTS HS1- Unable to create log file [/base path/WorkflowLogs/wf_11245.log.bin]. Cannot create log file in the specified location

CONFIG FILE :

####################################################################
#Input File Details
####################################################################
input
{
file
{
path =>"C:\Users\manid8\Desktop\Logstash\Logstash\informatica\informaticalog.log"
start_position => "beginning"
}
}
####################################################################

Processing the log file

####################################################################
filter
{
#########Pattern Matching################
grok
{
match =>["message","%{DATE:last_updated_time}%{GREEDYDATA:others}%{TYPE}%{SPACE}%{CUSTOMWORD:Error_code}%{SPACE}%{JOBID},%{SPACE}%{EXAMPLE:resource_id}%{GREEDYDATA:alert_title}%{NUMBER1}%{SPACE}%{GREEDYDATA:alert_description}%{NUMBER1}"]
add_tag => [ "SQLALERT_TAG" ]
}
grok
{
match =>["message","%{DATE:last_updated_time}%{GREEDYDATA:others}%{TYPE1}%{SPACE}%{CUSTOMWORD:Error_code}%{SPACE}%{JOBID},%{SPACE}%{EXAMPLE:resource_id}%{GREEDYDATA:alert_title}%{SPACE}%{NUMBER1}%{GREEDYDATA:alert_description}"]
add_tag => [ "SPACEALERT_TAG" ]
}
#########Parsing through Space Alerts for information processing#########
if "SPACEALERT_TAG" in [tags]
{
######### Parsing through Informatica Space Alert/Error#########
if [Error_code] == "[LM_2006]"
{
########## Deriving Metric count #########
#metrics
#{
# meter => [ "events" ]
# add_field => { "SPACEALERT_INFA_COUNT" => "%{[events][count]}" }
# add_field => { "S_METRIC_NAME" => "Space Issue with %{resource_id}" }
# add_field => { "S_Message" => "%{message}" }
# add_tag => "SPACEALERT_INFA_TAG"
#}
######### Adding custom fields #########
mutate
{
#add_field => { "S_METRIC_NAME" => "Space Issue with %{resource_id}" }
add_field => { "S_LAST_UPDATED_TIME" => "%{last_updated_time}" }
add_field => { "S_RESOURCE_ID" => "%{resource_id}" }
add_field => { "S_KAFKA_MSG" => "%{last_updated_time},%{resource_id},%{metric_name},%{SPACEALERT_INFA_COUNT}" }
#add_field => { "S_Message" => "%{message}" }
add_field => { "S_METRIC_COUNT_T1" => "%{[resource_id][count]}" }
add_field => { "S_METRIC_COUNT" => "%{SPACEALERT_INFA_COUNT}" }
add_tag => "SPACEALERT_INFA_TAG1"
}

	}
}

}
####################################################################

Output file post processing

####################################################################
output
{
if "SPACEALERT_INFA_TAG1" in [tags]
{
stdout
{
codec => line
{
format => "
#####################################################
Messages
#####################################################
%{message}
#####################################################
Space Alert_INFA_MUTATE2 , %{S_LAST_UPDATED_TIME},%{S_RESOURCE_ID},%{S_METRIC_NAME},%{SPACEALERT_INFA_COUNT},%{S_METRIC_COUNT_T1}
#####################################################
Kafka Message , %{S_KAFKA_MSG}
"
}
}

}

if "SPACEALERT_TAG" in [tags]
{
	if [Error_code] == "[LM_2006]"
	{
		stdout 
		{
			codec => line
			{
				format => "
				#####################################################
				Messages
				#####################################################
				%{message}
				#####################################################
				Space Alert_INFA_MUTATE2 , %{S_LAST_UPDATED_TIME},%{S_RESOURCE_ID},%{S_METRIC_NAME},%{SPACEALERT_INFA_COUNT},%{S_METRIC_COUNT_T1}
				#####################################################
				Kafka Message , %{S_KAFKA_MSG}
				"
			}
		}
	}
}

}

if "SPACEALERT_INFA_MUTATE_TAG" in [tags]
{
		stdout 
		{
			codec => line
			{
				format => "
				#####################################################
				Messages2
				#####################################################
				%{message}
				#####################################################
				Space Alert_INFA_MUTATE3 , %{S_LAST_UPDATED_TIME},%{S_RESOURCE_ID},%{S_METRIC_NAME},%{SPACEALERT_INFA_COUNT},%{S_METRIC_COUNT_T1}
				#####################################################
				Kafka Message2 , %{S_KAFKA_MSG}
				#####################################################
				Space Alert_INFA_MUTATE4 , %{S_LAST_UPDATED_TIME},%{S_RESOURCE_ID},%{S_METRIC_NAME},%{SPACEALERT_INFA_COUNT},%{[SPACEALERT_INFA_EVENT][count]}
				"
			}
		}
}

(system) #2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.