Logstash Grok Filter Issue

[peter.alfafara]

Hi,

Can anyone help us in creating the proper filter for our logs. See sample below as we see in Kibana:

_@timestamp:March 17th 2017, 23:22:49.774 port:52,718 @version:1 host:127.0.0.1 message:{"client":{"name":"Client","address":"127.0.0.1","subscriptions":["ALL","client:Client"],"version":"0.28.4","timestamp":1489764153},"check":{"command":"/opt/sensu/embedded/bin/check-memory.rb","interval":60,"standalone":true,"handler":"event_stream","subscribers":["ALL"],"name":"check_memory","issued":1489764169,"executed":1489764169,"duration":0.109,"output":"MEM OK - free system memory: 7541 MB\n","status":0,"type":"standard","history":["0","0","0","0","0","0","0","0","0","0","0","0","0","0","0","0","0","0","0","0","0"],"total_state_change":0},"occurrences":1,"occurrences_watermark":1,"action":"create","timestamp":1489764169,"id":"044e51bf-8a90-4df8-9e73-15a7a2e5c0c1","last_ok":1489764169,"silenced":false,"silenced_by":[]} tags:_grokparsefailure _id:AVrc3ThKoYxWJ5cxJt6c type:log

What we want to achieve is to remove all fields except for " message" and "timestamp" and modify the message field to look something like
clientName: "Client" clientAddress: "127.0.0.1" checkCommand: "/opt/sensu/embedded/bin/check-memory.rb" checkOutput: "MEM OK - free system memory: 7541 MB\n"

Below is our current conf file

input {
tcp {
port => 5514
}
}

filter {
grok {
match => { "message" => "clientName %{WORD:[client][name]} clientAddress %{IP:[client][address]}
checkThresholds %{DATA:[check][thresholds]} checkOutput %{DATA:[check][output]}}
}

if "Sensu" in [message] {
drop { }
}

mutate {
remove_field => [ "port", "@version", "host", [check][history], "tags", "_id", "_type", "_index"]
}
}

output {
elasticsearch {
hosts => "http://localhost:9200"
}
}

Hope you can help. Thank you

[YuWatanabe]

@peter.alfafara

Since the message looks like a json format message , if I were you , I would parse the field with json filter first and then do some extra working with mutate filter .

Like,

filter {
  json { source => "message" }

  mutate {
    add_filed => {"clientName" => "%{[client][name] ...}"}
    remove_filed =>["client"...]
  }
}