Remove message on filter before inserting in ES

Elastic Stack Logstash
1

Hi,

Here is my actual config and output :

filter {

if [type] == "ufdb" {

            grok {
                    match => [  "message", "%{TIMESTAMP_ISO8601:logdate}%{SPACE}%{DATA}%{SPACE}%{WORD:status}%{SPACE}%{USERNAME:group}%{SPACE}%{IP:ip}%{SPACE}%{DATA}%{SPACE}%{WORD:category}%{SPACE}%{URI:url} %{WORD:method}" ]

                    }

            date {
      match => [ "logdate", "ISO8601", "YYYY-MM-dd HH:mm:ss" ]
      target => "logdate"
      locale => "en"
    }
            mutate {
                    remove => [ 'file','offset','host' ]
  }

}

output on ES :

"hits": {
"total": 8165,
"max_score": 1,
"hits": [
{
"_index": "shit",
"_type": "ufdb",
"_id": "AVBCidxwvfdc1_d4SQML",
"_score": 1,
"_source": {
"message": "2015-11-30 15:56:06 [2785] PASS level3 10.249.10.91 level3 ads http://dm.ccmbg.com/28.... GET",
"@version": "1",
"@timestamp": "2015-10-07T13:42:20.958Z",
"type": "ufdb",
"logdate": "2015-11-30T14:56:06.000Z",
"status": "PASS",
"group": "level3",
"ip": "10.249.10.91",
"category": "ads",
"url": "http://dm.ccmbg.com/28....",
"method": "GET"
}
},

On my output I want to remove the message field like I have done for "file,host,offset"

When I do a remove or remove_field, I just don’t get any data in my index.

Is it possible to remove the message ?

Here is the output I want :

  "hits": {
  "total": 8165,
  "max_score": 1,
  "hits": [
     {
        "_index": "shit",
        "_type": "ufdb",
        "_id": "AVBCidxwvfdc1_d4SQML",
        "_score": 1,
        "_source": {
           "@version": "1",
           "@timestamp": "2015-10-07T13:42:20.958Z",
           "type": "ufdb",
           "logdate": "2015-11-30T14:56:06.000Z",
           "status": "PASS",
           "group": "level3",
           "ip": "10.249.10.91",
           "category": "ads",
           "url": "http://dm.ccmbg.com/28....",
           "method": "GET"
        }
     },

Any ideas ?

2

message isn't different from any other field and you can remove it as you please. Removal of that field will not stop the remains of the message to be sent to ES.

3

Thanks but there when I put ''message'' in the remove parameter, my filter don't work at all. I guess the grok don't not match the log line.

4

It sounds like you're removing the message field before the grok filter that attempts to parse that field. Filters are evaluated in order and if you have multiple configuration files in /etc/logstash/conf.d they're read in alphabetical order.