How to use filter exception syslog messages using logstash

Hi All,

I have got below from email alert

I want to setup my email exception messages with few lines. So, that developer can check why exception trigger

Exception Alert

@timestamp: 2021-06-08T10:41:24.570Z
_id: Sm4263kBz7SOxaiwcp_J
_index: logstash-2021.06.08
_type: _doc
message: 2021-06-08 10:17:23.624 INFO 1 --- [ container-172] c.f.f.s.p.r.TestCaseResponseProcessor : Saving TestCaseResponses in ES"}
{"@timestamp":"2021-06-08T10:17:23+00:00","type":"syslog_json","tag":"prod_fx-control-plane.1.jau26s2a","relayhost":"[163.107.196.104.bc.googleusercontent.com](http://163.107.196.104.bc.googleusercontent.com/)","relayip":"104.196.107.163","logsource":"fx-prod-1","hostname":"fx-prod-1","program":"prod_fx-control-plane.1.jau26s2a","priority":"30","severity":"6","facility":"3","severity_label":"info","facility_label":"daemon","procid":"-","message":"<30>Jun 8 10:17:23 fx-prod-1 prod_fx-control-plane.1.jau26s2a 2021-06-08 10:17:23.654 INFO 1 --- [ container-172] c.f.f.s.p.r.TestCaseResponseProcessor : Reviewing TestCaseResponses Status"}
{"@timestamp":"2021-06-08T10:17:23+00:00","type":"syslog_json","tag":"prod_fx-control-plane.1.jau26s2a","relayhost":"[163.107.196.104.bc.googleusercontent.com](http://163.107.196.104.bc.googleusercontent.com/)","relayip":"104.196.107.163","logsource":"fx-prod-1","hostname":"fx-prod-1","program":"prod_fx-control-plane.1.jau26s2a","priority":"30","severity":"6","facility":"3","severity_label":"info","facility_label":"daemon","procid":"-","message":"<30>Jun 8 10:17:23 fx-prod-1 prod_fx-control-plane.1.jau26s2a 2021-06-08 10:17:23.654 INFO 1 --- [ container-172] c.f.f.s.p.r.TestCaseResponseProcessor : Update Review Status for HoursExceptionsApprovalHistoryGetAllowedRbac"}
{"@timestamp":"2021-06-08T10:17:23+00:00","type":"syslog_json","tag":"prod_fx-control-plane.1.jau26s2a","relayhost":"[163.107.196.104.bc.googleusercontent.com](http://163.107.196.104.bc.googleusercontent.com/)","relayip":"104.196.107.163","logsource":"fx-prod-1","hostname":"fx-prod-1","program":"prod_fx-control-plane.1.jau26s2a","priority":"30","severity":"6","facility":"3","severity_label":"info","facility_label":"daemon","procid":"-","message":"<30>Jun 8 10:17:23 fx-prod-1 prod_fx-control-plane.1.jau26s2a 2021-06-08 10:17:23.683 INFO 1 --- [ container-172] c.f.f.s.p.r.TestCaseResponseProcessor : Processing AutoSuggestion for HoursExceptionsApprovalHistoryGetAllowedRbac"}
{"@timestamp":"2021-06-08T10:17:23+00:00","type":"syslog_json","tag":"prod_fx-control-plane.1.jau26s2a","relayhost":"[163.107.196.104.bc.googleusercontent.com](http://163.107.196.104.bc.googleusercontent.com/)","relayip":"104.196.107.163","logsource":"fx-prod-1","hostname":"fx-prod-1","program":"prod_fx-control-plane.1.jau26s2a","priority":"30","severity":"6","facility":"3","severity_label":"info","facility_label":"daemon","procid":"-","message":"<30>Jun 8 10:17:23 fx-prod-1 prod_fx-control-plane.1.jau26s2a 2021-06-08 10:17:23.710 INFO 1 --- [ container-172] c.f.f.s.p.r.TestCaseResponseProcessor : Processed AutoSuggestion for HoursExceptionsApprovalHistoryGetAllowedRbac"}
{"@timestamp":"2021-06-08T10:17:23+00:00","type":"syslog_json","tag":"prod_fx-control-plane.1.jau26s2a","relayhost":"[163.107.196.104.bc.googleusercontent.com](http://163.107.196.104.bc.googleusercontent.com/)","relayip":"104.196.107.163","logsource":"fx-prod-1","hostname":"fx-prod-1","program":"prod_fx-control-plane.1.jau26s2a","priority":"30","severity":"6","facility":"3","severity_label":"info","facility_label":"daemon","procid":"-","message":"<30>Jun 8 10:17:23 fx-prod-1 prod_fx-control-plane.1.jau26s2a 2021-06-08 10:17:23.710 INFO 1 --- [ container-172] c.f.f.s.p.r.TestCaseResponseProcessor : Updating Run [8a8093e379e5ef320179eb1229873475] with Validations"}
{"@timestamp":"2021-06-08T10:17:23+00:00","type":"syslog_json","tag":"prod_fx-control-plane.1.jau26s2a","relayhost":"[163.107.196.104.bc.googleusercontent.com](http://163.107.196.104.bc.googleusercontent.com
[107.196.104.bc.googleusercontent.com](http://163.107.196.104.bc.googleusercontent.com/)","relayip":"104.196.107.163","logsource":"fx-prod-1","hostname":"fx-prod-1","program":"prod_fx-control-plane.1.jau26s2a","priority":"30","severity":"6","facility":"3","severity_label":"info","facility_label":"daemon","procid":"-","message":"<30>Jun 8 10:17:23 fx-prod-1 prod_fx-control-plane.1.jau26s2a 2021-06-08 10:17:23.710 INFO 1 --- [ container-172] c.f.f.s.p.r.TestCaseResponseProcessor : Processed AutoSuggestion for HoursExceptionsApprovalHistoryGetAllowedRbac"}
{"@timestamp":"2021-06-08T10:17:23+00:00","type":"syslog_json","tag":"prod_fx-control-plane.1.jau26s2a","relayhost":"[163.107.196.104.bc.googleusercontent.com](http://163.107.196.104.bc.googleusercontent.com/)","relayip":"104.196.107.163","logsource":"fx-prod-1","hostname":"fx-prod-1","program":"prod_fx-control-plane.1.jau26s2a","priority":"30","severity":"6","facility":"3","severity_label":"info","facility_label":"daemon","procid":"-","message":"<30>Jun 8 10:17:23 fx-prod-1 prod_fx-control-plane.1.jau26s2a 2021-06-08 10:17:23.710 INFO 1 --- [ container-172] c.f.f.s.p.r.TestCaseResponseProcessor : Updating Run [8a8093e379e5ef320179eb1229873475] with Validations"}

And Here is the logstash.conf file

input {	
file {
    type => "java"
    path => "/elk/spring-boot-elk.log"
    codec => multiline {
    pattern => "^%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}.*"
    negate => "true"
    what => "previous"
    }
  }
}
 
filter {
  #If log line contains tab character followed by 'at' then we will tag that entry as stacktrace
  if [message] =~ "\tat" {
    grok {
      match => ["message", "^(\tat)"]
      add_tag => ["stacktrace"]
    }
  }
 
}
 
output {
   
  stdout {
    codec => rubydebug
  }
 
  # Sending properly parsed log events to elasticsearch
  elasticsearch {
    hosts => ["elasticsearch:9200"]
  }
}

Just tell me How can the message show in one line
HOW TO USE FILTER'S ?

Sorry, we don't support Elastalert here. You might consider using Kibana Alerting instead, which we do support.

forget about elastalert just tell me about logstash

It is not clear what you want to do.

What you want to filter?

You are sharing two different things, your alert e-mail queries a data that already has passed through logstash, you need to share your original data and explain what you want to do.

But, you will probably need to change your elastalert query, as is this query that creates the data that you send using e-mail.

1 Like

Ok, let me explain u

Previous we are triggering email from elalstalert with a small exception in rules.yml file in kibana UI under elastalert plugin. Exception means triggering an issue or error from my production server. So its generating small message see below

Exception Alert

@timestamp: 2021-06-01T11:59:38Z
_id: 6N5yx3kBz7SOxaiwHV-c
_index: logstash-2021.06.01
_type: _doc
message: <30>Jun  1 11:59:38 fx-prod-1 prod_fx-control-plane.1.8rfrnvi1 org.springframework.dao.CannotAcquireLockException: could not execute statement; SQL [n/a]; nested exception is org.hibernate.exception.LockAcquisitionException: could not execute statement
num_hits: 1
num_matches: 1

But my team want more error lines to be add in that exception so i search in google and found filter use in logstash after add that filter in logstash.conf file its generate which u can read above matter i have mention in my question so that generating alots of message . I want single message with few 10 or 15lines of message should show in my exception. Right now its showing only 4 or 7 lines.

Let me know

So this is indeed a question on Elastalert. :ok_hand:

This is related to elastalert, you are showing an example of a message after it was processed by logstash and indexed in elasticsearch.

Logstash filter are applied while processing messages, if you want to change your logstash pipeline you need to share examples of your messages before they are indexed, the messages that are inserted in logstash.

The message show by your alert and the message processed by logstash are two different things, if you want to show more messages in your alert you need to change your elastalert filter, if you want to change how your messages are indexed by logstash then you need to provide more information about the source of those messages and how you are indexing them.

Ok, let me explain u

Below is the exception message

Exception Alert

@timestamp: 2021-06-01T11:59:38Z
_id: 6N5yx3kBz7SOxaiwHV-c
_index: logstash-2021.06.01
_type: _doc
message: <30>Jun  1 11:59:38 fx-prod-1 prod_fx-control-plane.1.8rfrnvi1 org.springframework.dao.CannotAcquireLockException: could not execute statement; SQL [n/a]; nested exception is org.hibernate.exception.LockAcquisitionException: could not execute statement
num_hits: 1
num_matches: 1

But now my team want some more error of lines need to be add like example

Exception Alert

@timestamp: 2021-06-01T11:59:38Z
_id: 6N5yx3kBz7SOxaiwHV-c
_index: logstash-2021.06.01
_type: _doc
message: <30>Jun  1 11:59:38 fx-prod-1 prod_fx-control-plane.1.8rfrnvi1 org.springframework.dao.CannotAcquireLockException: could not execute statement; SQL [n/a]; nested exception is org.hibernate.exception.LockAcquisitionException: could not execute statement "HERE NEED SOME MORE LINES OF ERROR SHOULD COME"
num_hits: 1
num_matches: 1

I google and found some filter to use in logstash.conf file

input {
	tcp {
		port => 5044
#                codec => "json"
                codec => multiline {
                pattern => "^%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}.*"
                negate => "true"
                what => "previous"
             }
     }
}

filter {
  #If log line contains tab character followed by 'at' then we will tag that entry as stacktrace
  if [message] =~ "\tat" {
    grok {
      match => ["message", "^(\tat)"]
      add_tag => ["stacktrace"]
    }
  } 
}

## Add your filters / logstash plugins configuration here

output {
	elasticsearch {
		hosts => "elasticsearch:9200"
                user => "elastic"
                password => "kibana@1234"
	}
#        stdout { codec => rubydebug }
}

after i add filter, codec, pattern, negate, what in logstash.conf file

now getting exception message like this

@timestamp: 2021-06-08T10:41:24.570Z
_id: Sm4263kBz7SOxaiwcp_J
_index: logstash-2021.06.08
_type: _doc
message: 2021-06-08 10:17:23.624 INFO 1 --- [ container-172] c.f.f.s.p.r.TestCaseResponseProcessor : Saving TestCaseResponses in ES"}
{"@timestamp":"2021-06-08T10:17:23+00:00","type":"syslog_json","tag":"prod_fx-control-plane.1.jau26s2a","relayhost":"[163.107.196.104.bc.googleusercontent.com](http://163.107.196.104.bc.googleusercontent.com/)","relayip":"104.196.107.163","logsource":"fx-prod-1","hostname":"fx-prod-1","program":"prod_fx-control-plane.1.jau26s2a","priority":"30","severity":"6","facility":"3","severity_label":"info","facility_label":"daemon","procid":"-","message":"<30>Jun 8 10:17:23 fx-prod-1 prod_fx-control-plane.1.jau26s2a 2021-06-08 10:17:23.654 INFO 1 --- [ container-172] c.f.f.s.p.r.TestCaseResponseProcessor : Reviewing TestCaseResponses Status"}
{"@timestamp":"2021-06-08T10:17:23+00:00","type":"syslog_json","tag":"prod_fx-control-plane.1.jau26s2a","relayhost":"[163.107.196.104.bc.googleusercontent.com](http://163.107.196.104.bc.googleusercontent.com/)","relayip":"104.196.107.163","logsource":"fx-prod-1","hostname":"fx-prod-1","program":"prod_fx-control-plane.1.jau26s2a","priority":"30","severity":"6","facility":"3","severity_label":"info","facility_label":"daemon","procid":"-","message":"<30>Jun 8 10:17:23 fx-prod-1 prod_fx-control-plane.1.jau26s2a 2021-06-08 10:17:23.654 INFO 1 --- [ container-172] c.f.f.s.p.r.TestCaseResponseProcessor : Update Review Status for HoursExceptionsApprovalHistoryGetAllowedRbac"}
{"@timestamp":"2021-06-08T10:17:23+00:00","type":"syslog_json","tag":"prod_fx-control-plane.1.jau26s2a","relayhost":"[163.107.196.104.bc.googleusercontent.com](http://163.107.196.104.bc.googleusercontent.com/)","relayip":"104.196.107.163","logsource":"fx-prod-1","hostname":"fx-prod-1","program":"prod_fx-control-plane.1.jau26s2a","priority":"30","severity":"6","facility":"3","severity_label":"info","facility_label":"daemon","procid":"-","message":"<30>Jun 8 10:17:23 fx-prod-1 prod_fx-control-plane.1.jau26s2a 2021-06-08 10:17:23.683 INFO 1 --- [ container-172] c.f.f.s.p.r.TestCaseResponseProcessor : Processing AutoSuggestion for HoursExceptionsApprovalHistoryGetAllowedRbac"}
{"@timestamp":"2021-06-08T10:17:23+00:00","type":"syslog_json","tag":"prod_fx-control-plane.1.jau26s2a","relayhost":"[163.107.196.104.bc.googleusercontent.com](http://163.107.196.104.bc.googleusercontent.com/)","relayip":"104.196.107.163","logsource":"fx-prod-1","hostname":"fx-prod-1","program":"prod_fx-control-plane.1.jau26s2a","priority":"30","severity":"6","facility":"3","severity_label":"info","facility_label":"daemon","procid":"-","message":"<30>Jun 8 10:17:23 fx-prod-1 prod_fx-control-plane.1.jau26s2a 2021-06-08 10:17:23.710 INFO 1 --- [ container-172] c.f.f.s.p.r.TestCaseResponseProcessor : Processed AutoSuggestion for HoursExceptionsApprovalHistoryGetAllowedRbac"}
{"@timestamp":"2021-06-08T10:17:23+00:00","type":"syslog_json","tag":"prod_fx-control-plane.1.jau26s2a","relayhost":"[163.107.196.104.bc.googleusercontent.com](http://163.107.196.104.bc.googleusercontent.com/)","relayip":"104.196.107.163","logsource":"fx-prod-1","hostname":"fx-prod-1","program":"prod_fx-control-plane.1.jau26s2a","priority":"30","severity":"6","facility":"3","severity_label":"info","facility_label":"daemon","procid":"-","message":"<30>Jun 8 10:17:23 fx-prod-1 prod_fx-control-plane.1.jau26s2a 2021-06-08 10:17:23.710 INFO 1 --- [ container-172] c.f.f.s.p.r.TestCaseResponseProcessor : Updating Run [8a8093e379e5ef320179eb1229873475] with Validations"}
{"@timestamp":"2021-06-08T10:17:23+00:00","type":"syslog_json","tag":"prod_fx-control-plane.1.jau26s2a","relayhost":"[163.107.196.104.bc.googleusercontent.com](http://163.107.196.104.bc.googleusercontent.com
[107.196.104.bc.googleusercontent.com](http://163.107.196.104.bc.googleusercontent.com/)","relayip":"104.196.107.163","logsource":"fx-prod-1","hostname":"fx-prod-1","program":"prod_fx-control-plane.1.jau26s2a","priority":"30","severity":"6","facility":"3","severity_label":"info","facility_label":"daemon","procid":"-","message":"<30>Jun 8 10:17:23 fx-prod-1 prod_fx-control-plane.1.jau26s2a 2021-06-08 10:17:23.710 INFO 1 --- [ container-172] c.f.f.s.p.r.TestCaseResponseProcessor : Processed AutoSuggestion for HoursExceptionsApprovalHistoryGetAllowedRbac"}
{"@timestamp":"2021-06-08T10:17:23+00:00","type":"syslog_json","tag":"prod_fx-control-plane.1.jau26s2a","relayhost":"[163.107.196.104.bc.googleusercontent.com](http://163.107.196.104.bc.googleusercontent.com/)","relayip":"104.196.107.163","logsource":"fx-prod-1","hostname":"fx-prod-1","program":"prod_fx-control-plane.1.jau26s2a","priority":"30","severity":"6","facility":"3","severity_label":"info","facility_label":"daemon","procid":"-","message":"<30>Jun 8 10:17:23 fx-prod-1 prod_fx-control-plane.1.jau26s2a 2021-06-08 10:17:23.710 INFO 1 --- [ container-172] c.f.f.s.p.r.TestCaseResponseProcessor : Updating Run [8a8093e379e5ef320179eb1229873475] with Validations"}
{"@timestamp":"2021-06-10T08:11:40+00:00","type":"syslog_json","tag":"prod_fx-control-plane.1.jau26s2a","relayhost":"163.107.196.104.bc.googleusercontent.com","relayip":"104.196.107.163","logsource":"fx-prod-1","hostname":"fx-prod-1","program":"prod_fx-control-plane.1.jau26s2a","priority":"30","severity":"6","facility":"3","severity_label":"info","facility_label":"daemon","procid":"-","message":"<30>Jun 10 08:11:40 fx-prod-1 prod_fx-control-plane.1.jau26s2a #011at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:1476) ~[spring-rabbit-2.2.1.RELEASE.jar!\/:2.2.1.RELEASE]"}
{"@timestamp":"2021-06-10T08:11:40+00:00","type":"syslog_json","tag":"prod_fx-control-plane.1.jau26s2a","relayhost":"163.107.196.104.bc.googleusercontent.com","relayip":"104.196.107.163","logsource":"fx-prod-1","hostname":"fx-prod-1","program":"prod_fx-control-plane.1.jau26s2a","priority":"30","severity":"6","facility":"3","severity_label":"info","facility_label":"daemon","procid":"-","message":"<30>Jun 10 08:11:40 fx-prod-1 prod_fx-control-plane.1.jau26s2a #011at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:1467) ~[spring-rabbit-2.2.1.RELEASE.jar!\/:2.2.1.RELEASE]"}

Unnecessary contain is showing in exception message i only want to have few lines of exception message should show