Logstash grep and grok

[tarunmendon]

I am new to Elasticstack. I am trying to implement a logstash pipeline in which the a file would be processed and it would filter(grep) and output if the line of file contains following keyword -

1. java.lang.Exception - Any line of file containing Exception should be filtered and be available on Kibana
2. XYZ process completed.
   I tried following but it seems to outputting all the contents that do not match the Exception too.

input {
  beats {
    port => 5044
    tags => "exception"
  }
}
filter{
  if "exception" in [tags]{
    grok {
    match => { message => "Exception"
    }
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}" 
  }
}

Please help and advise.

[grumo35]

Add the following condition in the output maybe ?

output {
 if "exception" in [tags] {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}" 
  }
}
}

[Badger]

You could try

filter{
}
output {
  if "Exception" in [message] or "XYZ process completed" in [message] {
    elasticsearch {
      hosts => ["http://localhost:9200"]
      index => "%{[@metadata][beat]}-%{[@metadata][version]}" 
    }
  }
}

[tarunmendon]

Hello @Badger/@grumo35,
Thanks for the reply. I added following in the pipeline.yml -

input {
  beats {
    port => 5044
    id=> "filebeat_plugin"
    tags => "exception"
  }
}
filter{
 }
output {
if "Exception" in [message] or "XYZ process completed" in [message]{
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}" 
  }
  }
}

I added following in the log :
java.lang.NumberFormatException
XYZ process completed
java.lang.OtherException

Kibana output didnot process the "XYZ process completed". PFA attachment. Any idea about it?