Logstash how to output the source log path in my message


(fernando) #1

Im trying to get the number of times an ERROR|Error|error or Warn loglevel shows up in my log file and depending of the amount of times it repeats send a email/slack to the service owner with a summary like that

CDP-error: 5 Errors on file XXXX.log in the last 3 hours
CDP-Warn: 10 Warning on file XXXX.log in the last 3 hours

Right now Im just testing but I dont know why seem not to work properly, Im able to get the count when error or Warn appears but Im not able to add the path of the file where this error is happening.

Im sending my error messages using echo and manually pushing it to the log file:

echo "18/10/31 12:01:54 ERROR netty.Dispatcher: Message RemoteProcessDisconnected(10.192.64.33:56936) dropped. Could not find BlockManagerEndpoint1." >> OptusCatPush_test_async_6.log

Here is my output:

Settings: Default pipeline workers: 80
Pipeline main started
{
      "@version" => "1",
    "@timestamp" => "2019-02-12T00:15:15.242Z",
       "message" => "dm0010.hkg1.turn.com",
       "w_level" => {
           "count" => 1,
         "rate_1m" => 0.0,
         "rate_5m" => 0.0,
        "rate_15m" => 0.0
    },
          "tags" => [
        [0] "metric_a"
    ]
}
{
      "@version" => "1",
    "@timestamp" => "2019-02-12T00:15:30.240Z",
       "message" => "dm0010.hkg1.turn.com",
       "w_level" => {
           "count" => 1,
         "rate_1m" => 0.0,
         "rate_5m" => 0.0,
        "rate_15m" => 0.0
    },
          "tags" => [
        [0] "metric_a"
    ]
}

or

Pipeline main started
{
      "@version" => "1",
    "@timestamp" => "2019-02-12T00:08:35.885Z",
       "message" => "dm0010.hkg1.turn.com",
       "w_level" => {
           "count" => 1,
         "rate_1m" => 0.0,
         "rate_5m" => 0.0,
        "rate_15m" => 0.0
    },
          "tags" => [
        [0] "metric_a"
    ]
}
CDP-ERROR: {"count":1,"rate_1m":0.0,"rate_5m":0.0,"rate_15m":0.0}
{
      "@version" => "1",
    "@timestamp" => "2019-02-12T00:08:50.886Z",
       "message" => "dm0010.hkg1.turn.com",
       "w_level" => {
           "count" => 1,
         "rate_1m" => 0.0,
         "rate_5m" => 0.0,
        "rate_15m" => 0.0
    },
          "tags" => [
        [0] "metric_a"
    ]
}
CDP-ERROR: {"count":1,"rate_1m":0.0,"rate_5m":0.0,"rate_15m":0.0}
{
      "@version" => "1",
    "@timestamp" => "2019-02-12T00:09:05.885Z",
       "message" => "dm0010.hkg1.turn.com",
       "w_level" => {
           "count" => 6,
         "rate_1m" => 0.0,
         "rate_5m" => 0.0,
        "rate_15m" => 0.0
    },
          "tags" => [
        [0] "metric_a"
    ]
}

Here is my config:

input {
  file {
    path => [ '/home/fvega/OptusCatPush_test_async_6.log' ]
    type => "oozie"
  }
}

filter {
  if [type] == "oozie" {
    grok {
      match => [ "message", "%{YEAR:year}\/%{MONTHNUM:month}\/%{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second} %{LOGLEVEL:loglevel} %{GREEDYDATA:message}" ]
    }
  }

  if [loglevel] =~ /"|WARN|Warn|warn|"/ {
    metrics {
      meter => "w_level"
      flush_interval => 5
      clear_interval =>10
      add_tag => "metric_a"
    }
  }
  else if [loglevel] =~ /"|ERROR|Error|error|"/ {
    metrics {
      meter => "e_level"
      flush_interval => 5
      clear_interval =>10
      add_tag => "metric_b"
    }
  }
}
output {
  if "metric_a" in [tags] and [w_level] {
    stdout {
      codec => line {
        format => "CDP-WARN: %{[w_level][count]} in file %{path}"
      }
    }
  }
  else if "metric_b" in [tags] and [e_level] {
    stdout {
      codec => line {
         format => "CDP-ERROR: %{[e_level][count]} in file %{path}"
      }
    }
  }
}

and send the log file to Elastic as default. Not included here yet