Parsing Varying Logs with Logstash

Looking for some guidance on where to look for an example of how to do this properly. We have maillogs and postfix logs going into the same log file and I need to learn how to properly differentiate the logs.

So far these are the search patterns in grok that work, however, some of them are the same up to a certain point, and then there is examples where after that point, there either is, or isn't more entries:

  else if ([fields][log_type] == "maillog") {
    mutate {
      remove_field => [ "[host]" ]
    }
    mutate {
      add_field => {
        "host" => "%{[beat][hostname]}"
      }
    }
    grok {
      match =>
      [
      "message", "%{SYSLOGTIMESTAMP:log_server_timestamp} %{WORD:server} %{NOTSPACE:protocol}: %{WORD:action}, %{NOTSPACE}\[%{IP:ip6}:%{IP:ip}]",
      "message", "%{SYSLOGTIMESTAMP:log_server_timestamp} %{WORD:server} %{NOTSPACE:protocol}: %{WORD:action}, %{NOTSPACE}=%{NOTSPACE:user}, %{NOTSPACE}\[%{IP:ip6}:%{IP:ip}]",
      "message", "%{SYSLOGTIMESTAMP:log_server_timestamp} %{WORD:server} %{NOTSPACE:protocol}: %{WORD:action}, %{NOTSPACE}=%{NOTSPACE:user}, %{NOTSPACE}\[%{IP:ip6}:%{IP:ip}], %{NOTSPACE}\[%{INT:port}\]",
      "message", "%{SYSLOGTIMESTAMP:log_server_timestamp} %{WORD:server} %{NOTSPACE:protocol}: %{WORD:action}, %{NOTSPACE}=%{NOTSPACE:user}, %{NOTSPACE}\[%{IP:ip6}:%{IP:ip}], %{NOTSPACE}\[%{INT:port}\], %{NOTSPACE}=%{INT:top}, %{NOTSPACE}=%{INT:retr}, %{NOTSPACE}=%{INT:rcvd}, %{NOTSPACE}=%{INT:sent}, %{NOTSPACE}=%{INT:time}",
      "message", "%{SYSLOGTIMESTAMP:log_server_timestamp} %{WORD:server} %{NOTSPACE:protocol}: %{WORD:action}, %{NOTSPACE}=%{NOTSPACE:user}, %{NOTSPACE}\[%{IP:ip6}:%{IP:ip}], %{NOTSPACE}=%{INT:headers}, %{NOTSPACE}=%{INT:body}, %{NOTSPACE}=%{INT:rcvd}, %{NOTSPACE}=%{INT:sent}, %{NOTSPACE}=%{INT:time}, %{NOTSPACE}=%{INT:starttls}"
      ]
    }
  }

So the obvious example is where sometimes the logs end at port number, but sometimes they include extra information after port number.

I am trying to find the most efficient way so that when there is more information after port number, it will continue parsing rather than stopping at port number.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.