Strange behavior in logstash after remove field message

[gbeltramelli]

Hi, first of all sorry for my english!
im using version 7.14 of ELK and i`m having some strange behavior when using
mutate { remove_field => [ "message" ] }

I have some logs from a DataPower then I use GROK to make them more usefull, without the mutate logs everything go fine but when I add that line some logs never appear in the Discover in Kibana.

this is my logstash

does anyone have any idea why when doing a remove_field logs never reach the elastic output? No error in but the way.

input {
  tcp {
    port => 8089
    tags => [datapower_log]
    codec => multiline {
      pattern => "<14>"
      negate => true
      what => "previous"
    }
 }
}

filter {
	if "INPUT" in [message] {
	    grok {
	     match => { "message" =>"%{NOTSPACE:Codigo}%{SYSLOGTIMESTAMP:fecha} %{GREEDYDATA:nodo}\[%{WORD:ResponseCode}\]\[%{WORD:debug}\]\[%{WORD:level}\]\ wsgw\(%{WORD:WS}\): trans\(%{WORD:trans}\)\[%{WORD:tipo}\]\[%{IP:cliente}\] gtid\(%{WORD:id}\): \nINPUT:(?m)%{GREEDYDATA:INPUT}\nOUTPUT:(?m)%{GREEDYDATA:OUTPUT}" }
	        }
	mutate {
	   remove_field => [ "message" ]
	}
	}
output {
  if "datapower_log" in [tags] {
    elasticsearch {
      hosts => ['https://xxxxxxxxx:9200']
      index => "dp-log-%{+YYYY.MM.dd}"
      cacert => '/etc/logstash/bps-net-cer-ca.pem'
      ssl => true
      ssl_certificate_verification => false
      user => elastic
      password => xxxxxxxxxxx
    }
   }
  stdout {
    codec => rubydebug { metadata => false }
  }
}

[ptamba]

if the event doesn’t match your grok, then it will be removed because of that mutate plugin

you can remove field when grok pattern matches event by doing

filter {
  grok { 
     match => { “message”, “<grok_pattern>” }
     remove_field => [“message”] 
  }
}

then the field will be removed only if the filter is successful. remove_field is one of filter common options that can be applied in any filter. more info here

Grok filter plugin | Logstash Reference [7.15] | Elastic