# Logstash : Obtain metrics from log file based on alert/error message
**URL:** https://discuss.elastic.co/t/logstash-obtain-metrics-from-log-file-based-on-alert-error-message/85239
**Category:** Logstash
**Created:** [May 10, 2017, 12:01pm UTC](https://discuss.elastic.co/t/logstash-obtain-metrics-from-log-file-based-on-alert-error-message/85239 "2017-05-10T12:01:30Z")
**Posts on this page:** 2
**Page:** 1
### Author:  [@Kavin\_MJR](https://discuss.elastic.co/u/Kavin_MJR)
#### Post date: [May 10, 2017, 12:01pm UTC](https://discuss.elastic.co/t/logstash-obtain-metrics-from-log-file-based-on-alert-error-message/85239/1 "2017-05-10T12:01:30Z")
We are trying to read a input log file. Assume that the log file is from an application and it might have a combination both alerts and errors. We are trying to separate the alert and error messages and compute metrics on the same.
Senario 1: If there is a space issue on the application and the space error log message appears twice in the log file. I m trying to obtain the following.
- timeStamp for the alert/error.
- Server name from the message
- Metric name eg: Space issue on the server
- Compute the count.
- We are able to get the alert/error out of the log and the 1st three fields.
- We are having trouble computing the count using metrics and combining this with the 1st 3 fields.
Output should look like,
"2017-01-17 22:15:43","RTS HS1","Cannot create log file in the specified location",2
Input File:
2017-01-17 22:11:42,846 INFO [Domain Monitor] [DOM\_10142] The update and timeout properties for the master gateway node is set in the domain configuration repository. The row update interval time is set to [8000] milliseconds and the out-of-date row timeout is set to [32000] milliseconds.
2017-01-17 22:12:42,007 INFO [Domain Monitor] [RR\_4035] SQL Alert [ORA-00911] INFA RTS Job5, RTS DB2- There is a problem connecting to your SQL database. This could be a temporary SQL alert network issue, or the schema changed in one of your tables.
2017-01-17 22:12:45,007 INFO [Domain Monitor] [RR\_4035] SQL Alert [ORA-00911] INFA RTS Job5, RTS DB2- There is a problem connecting to your SQL database. This could be a temporary SQL alert network issue, or the schema changed in one of your tables.
2017-01-17 22:14:43,060 INFO [Domain Monitor] [DOM\_10155] No master gateway found in the domain configuration repository. Setting this node to master gateway node.
2017-01-17 22:15:43,060 INFO [Domain Monitor] SPACE Alert [LM\_2006] INFA RTS Job1, RTS HS1- Unable to create log file [/base path/WorkflowLogs/wf\_11245.log.bin]. Cannot create log file in the specified location
2017-01-17 22:16:43,062 INFO [Domain Monitor] [DOM\_10152] Updated data row for node [node01\_host1].
2017-01-17 22:17:43,060 INFO [Domain Monitor] SPACE Alert [LM\_2006] INFA RTS Job1, RTS HS1- Unable to create log file [/base path/WorkflowLogs/wf\_11245.log.bin]. Cannot create log file in the specified location
2017-01-17 22:18:43,062 INFO [Domain Monitor] [DOM\_10152] Updated data row for node [node01\_host1].
2017-01-17 22:19:43,060 INFO [Domain Monitor] SPACE Alert [LM\_2006] INFA RTS Job1, RTS HS1- Unable to create log file [/base path/WorkflowLogs/wf\_11245.log.bin]. Cannot create log file in the specified location
CONFIG FILE :
####################################################################
#Input File Details
####################################################################
input
{
file
{
path =\>"C:\Users\manid8\Desktop\Logstash\Logstash\informatica\informaticalog.log"
start\_position =\> "beginning"
}
}
####################################################################
# Processing the log file
####################################################################
filter
{
#########Pattern Matching################
grok
{
match =\>["message","%{DATE:last\_updated\_time}%{GREEDYDATA:others}%{TYPE}%{SPACE}%{CUSTOMWORD:Error\_code}%{SPACE}%{JOBID},%{SPACE}%{EXAMPLE:resource\_id}%{GREEDYDATA:alert\_title}%{NUMBER1}%{SPACE}%{GREEDYDATA:alert\_description}%{NUMBER1}"]
add\_tag =\> ["SQLALERT\_TAG"]
}
grok
{
match =\>["message","%{DATE:last\_updated\_time}%{GREEDYDATA:others}%{TYPE1}%{SPACE}%{CUSTOMWORD:Error\_code}%{SPACE}%{JOBID},%{SPACE}%{EXAMPLE:resource\_id}%{GREEDYDATA:alert\_title}%{SPACE}%{NUMBER1}%{GREEDYDATA:alert\_description}"]
add\_tag =\> ["SPACEALERT\_TAG"]
}
#########Parsing through Space Alerts for information processing#########
if "SPACEALERT\_TAG" in [tags]
{
######### Parsing through Informatica Space Alert/Error#########
if [Error\_code] == "[LM\_2006]"
{
########## Deriving Metric count #########
#metrics
#{
# meter =\> ["events"]
# add\_field =\> { "SPACEALERT\_INFA\_COUNT" =\> "%{[events][count]}" }
# add\_field =\> { "S\_METRIC\_NAME" =\> "Space Issue with %{resource\_id}" }
# add\_field =\> { "S\_Message" =\> "%{message}" }
# add\_tag =\> "SPACEALERT\_INFA\_TAG"
#}
######### Adding custom fields #########
mutate
{
#add\_field =\> { "S\_METRIC\_NAME" =\> "Space Issue with %{resource\_id}" }
add\_field =\> { "S\_LAST\_UPDATED\_TIME" =\> "%{last\_updated\_time}" }
add\_field =\> { "S\_RESOURCE\_ID" =\> "%{resource\_id}" }
add\_field =\> { "S\_KAFKA\_MSG" =\> "%{last\_updated\_time},%{resource\_id},%{metric\_name},%{SPACEALERT\_INFA\_COUNT}" }
#add\_field =\> { "S\_Message" =\> "%{message}" }
add\_field =\> { "S\_METRIC\_COUNT\_T1" =\> "%{[resource\_id][count]}" }
add\_field =\> { "S\_METRIC\_COUNT" =\> "%{SPACEALERT\_INFA\_COUNT}" }
add\_tag =\> "SPACEALERT\_INFA\_TAG1"
}
```
}
}
```
}
####################################################################
# Output file post processing
####################################################################
output
{
if "SPACEALERT\_INFA\_TAG1" in [tags]
{
stdout
{
codec =\> line
{
format =\> "
#####################################################
Messages
#####################################################
%{message}
#####################################################
Space Alert\_INFA\_MUTATE2 , %{S\_LAST\_UPDATED\_TIME},%{S\_RESOURCE\_ID},%{S\_METRIC\_NAME},%{SPACEALERT\_INFA\_COUNT},%{S\_METRIC\_COUNT\_T1}
#####################################################
Kafka Message , %{S\_KAFKA\_MSG}
"
}
}
```
}
if "SPACEALERT_TAG" in [tags]
{
if [Error_code] == "[LM_2006]"
{
stdout
{
codec => line
{
format => "
#####################################################
Messages
#####################################################
%{message}
#####################################################
Space Alert_INFA_MUTATE2 , %{S_LAST_UPDATED_TIME},%{S_RESOURCE_ID},%{S_METRIC_NAME},%{SPACEALERT_INFA_COUNT},%{S_METRIC_COUNT_T1}
#####################################################
Kafka Message , %{S_KAFKA_MSG}
"
}
}
}
}
```
}
```
if "SPACEALERT_INFA_MUTATE_TAG" in [tags]
{
stdout
{
codec => line
{
format => "
#####################################################
Messages2
#####################################################
%{message}
#####################################################
Space Alert_INFA_MUTATE3 , %{S_LAST_UPDATED_TIME},%{S_RESOURCE_ID},%{S_METRIC_NAME},%{SPACEALERT_INFA_COUNT},%{S_METRIC_COUNT_T1}
#####################################################
Kafka Message2 , %{S_KAFKA_MSG}
#####################################################
Space Alert_INFA_MUTATE4 , %{S_LAST_UPDATED_TIME},%{S_RESOURCE_ID},%{S_METRIC_NAME},%{SPACEALERT_INFA_COUNT},%{[SPACEALERT_INFA_EVENT][count]}
"
}
}
}
```
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 7, 2017, 12:09pm UTC](https://discuss.elastic.co/t/logstash-obtain-metrics-from-log-file-based-on-alert-error-message/85239/2 "2017-06-07T12:09:19Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.