How to drop known errors in logstash

[psuvarna (Prakash Suvarna)]

Hello,
I am new to logstash and ELK stack. We are getting lot of errors in Kibana most of which are known to us. In order to concentrate on new errors while monitoring, we want a way to drop the known errors from logstash so that its not sent to Kibana dashboard.

Filter in our config file in logstash looks like this:

filter {

if [type] == "server_out_prod" {

    grok {
        match => { "message" => "\<%{DATA:msg_timestamp}\> \<%{DATA:msg_severity}\> \<%{DATA:msg_subsystem}\> \<%{DATA:msg_machine_name}\> \<%{DATA:msg_server_name}\> \<%{DATA:msg_thread_id}\> %{GREEDYDATA:msg_details}" }
		match => { "message" => "\<%{DATA:messageType}\>" }
		add_tag => [ "TM_%{messageType}" ]
    }
    
    mutate {
        add_field => { "project" => "estr_soa_prod" }
    }
	
	#if "TM_%{messageType}" in [tags] {
	#	drop {}
	#}  
}

}

The sample logfile looks something like this:

<Jun 19, 2018, 9:37:53,893 AM IST> <oracle.integration.platform.blocks.rest> <For service: integration/EventManagement!1.4_20180321*soa_dca150a3-b86f-44c2-bfb9-b417508355a3/PlayHoursRecordingCriteria Exception encountered for wsdl operation: GetPlayHoursRecordingCriteria corresponding to rest method: GET
oracle.fabric.common.BusinessFaultException: faultName: {{http://xmlns.oracle.com/EventManagement/EventManagement/PlayHoursRecordingCriteria}EventManagementFault}
messageType: {{http://xmlns.oracle.com/EventManagement/EventManagement/PlayHoursRecordingCriteria}GetPlayHoursRecordingCriteria_EventManagementFaultMessage}
parts: {{
fault_EventManagementFault=tns:CreationDateTime2018-06-19T09:37:53</tns:CreationDateTime>tns:Errortns:ErrorCodeEXC132</tns:ErrorCode>tns:DescriptionReturn-Code: X005; Message: Min/Max-Länge an Position 15 passt nicht</tns:Description></tns:Error>}

at oracle.fabric.CubeServiceEngine.getBusinessFault(CubeServiceEngine.java:2988)
at oracle.fabric.CubeServiceEngine.handleRequestResponseServerException(CubeServiceEngine.java:3910)
at oracle.fabric.CubeServiceEngine.request(CubeServiceEngine.java:653)
at oracle.integration.platform.blocks.mesh.SynchronousMessageHandler.doRequest(SynchronousMessageHandler.java:151)
at oracle.integration.platform.blocks.mesh.MessageRouter.request(MessageRouter.java:217)
at oracle.integration.platform.blocks.mesh.MeshImpl.request(MeshImpl.java:283)
at sun.reflect.GeneratedMethodAccessor2485.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318)
......

<Jun 19, 2018, 9:37:56,923 AM IST> <The size of the proposed value for context parameter oracle.soa.tracking.QuiescingOperationName is 30 and exceeds the permitted size of 26 for that parameter.>

Basically we want to drop this event whenever such an error occurs.

Appreciate your help on this.

Thanks,
Prakash.

[magnusbaeck (Magnus Bäck)]

Wrap a drop filter in a conditional that checks if the log entry matches the kind of messages you want to ignore. Something like this:

if [somefield] =~ /^The size of the proposed value/ {
  drop { }
}