How to remove unwanted tags from logstash

Elastic Stack Logstash
1

Hi i wanted to remove duplicate tags like ("@version" ,"beat","hostname","name" ) , how can i do that ?

    {
                    "msg" => "Beginning Product data refresh",
        "timestamp_match" => "2017-06-25 12:16:13,904",
                  "level" => "INFO",
             "input_type" => "log",
                 "source" => "/archives/logs/tomcat7-8090/download.log",
                "message" => "[2017-06-25 12:16:13,904]  :|:  INFO   :|:  lvprdsndlbfe1.lv.jabodo.com  :|:    :|:    :|:       :|:    :|:  c.m.c.PeriodicProductDataRefresher                            :|:   - Beginning Product data refresh",
                   "type" => "log",
                   "tags" => [
            [0] "multiline",
            [1] "beats_input_codec_multiline_applied"
        ],
             "@timestamp" => 2017-06-25T16:16:23.686Z,
               "@version" => "1",
                   "beat" => {
           "hostname" => "lvprdsndlbfe1",
             "name" => "lvprdsndlbfe1",
        "version" => "5.4.0"
        },
                  "class" => "c.m.c.PeriodicProductDataRefresher",
              "host_name" => "lvprdsndlbfe1.lv.jabodo.com"
    }
2

Use a mutate filter's remove_field option.

3

I am already using mutate to remove unwanted tabs and new lines in the input .

Do i have to use mutate after parsing grok ( i am just concern on logstash server )

input {
    beats {
    client_inactivity_timeout => 86400
    port => 5044
    codec => multiline {
      pattern => "^\[%{TIMESTAMP_ISO8601}\]"
      negate => true
      what => previous
     }
    }
}
filter {
#  csv {
#     separator => ":|:"
#     columns => ["Timestamp","Level","hostname","coidkey","Close","Volume", "Volume, "Currency","Weighted", "Price"]
#  }
#}

  mutate {
    gsub => [
      # replace all forward slashes with underscore
      #"fieldname", "/", "_",
      # replace backslashes, question marks, hashes, and minuses
      # with a dot "."
      #"fieldname2", "[\\?#-]", "."
      "message", "\t", " ",
      "message", "\n", " "
    ]
  }
    grok {
    match => { "message" => "\[%{TIMESTAMP_ISO8601:timestamp_match}\]%{SPACE}\:\|\:%{SPACE}%{WORD:level}%{SPACE}\:\|\:%{SPACE}%{USERNAME:host_name}%{SPACE}\:\|\:%{SPACE}%{GREEDYDATA:coidkey}%{SPACE}\:\|\:%{SPACE}%{GREEDYDATA:clientinfo}%{SPACE}\:
\|\:%{SPACE}(%{IP:clientip})?%{SPACE}\:\|\:%{SPACE}%{GREEDYDATA:Url}%{SPACE}\:\|\:%{SPACE}%{JAVACLASS:class}%{SPACE}\:\|\:%{SPACE}%{USER:ident}%{SPACE}%{GREEDYDATA:msg}"}   remove_field => [ "ident","offset","name","version","host" ]
   }
}
output {
    stdout { codec => rubydebug }

  if "_grokparsefailure" in [tags] {
    # write events that didn't match to a file
    file { "path" => "/tmp/grok_failures.txt" }
  } else{
     elasticsearch {
       hosts => "dfsyselastic.df.jabodo.com:9200"
       user => "UN"
       password => "PW"
       index => "vicinio-%{+YYYY.MM.dd}"
       document_type => "log"
     }
   }
}
4

I am already using mutate to remove unwanted tabs and new lines in the input .

That doesn't matter.

Do i have to use mutate after parsing grok ( i am just concern on logstash server )

The fields you were talking about exist before the grok filter so it doesn't matter if you delete them before your after the grok filter.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.