How to cut off the part of syslog

[YvesZhi]

I've some micro services, which are deployed with Docker. They send their logs to my Logstash with the log driver syslog. Here is the config of my Logstash:

input {
  syslog {
    port => 9771
    type => "syslog"
  }
}

filter {
}

output {
  file {
    path => "/var/log/logstash/%{+YYYY-MM-dd}/logstash-%{+HH}.log"
  }
  if "www.envoyproxy.io" in [message] {
    file {
      path => "/var/log/logstash/%{+YYYY-MM-dd}/test-%{+HH}.log"
      codec => line { format => "%{message}" }
    }
  }
}

I can see logs in the file test-01.log like this:

<30>Mar  8 11:42:45 2867370d06d5[8119]: 172.16.0.226,48982,envoy,2023-03-08T03:42:39.340Z,34.142.199.10,443,-,www.envoyproxy.io,HTTP/1.1,GET,/,HTTP/1.1,200,17304,2023-03-08T03:42:39.340Z,17304,-,-,curl/7.29.0,-,1000,0,0,0,0,0,0,0,0,0,0,0

I want to remove the part <30>Mar 8 11:42:45 2867370d06d5[8119]: before writing the logs into the file test-01.log but I don't know how.

[Rios]

Welcome to the comunity!

You can use something like this:

filter {
     grok {
       match => {  "message" => "<%{NONNEGINT:syslog_abspri}>%{SYSLOGTIMESTAMP:timestamp} %{PROG:program}\[%{POSINT:pid}\]: %{GREEDYDATA:message}" }
       overwrite => [ "message" ]
     }
}

Result:

{
              "pid" => "8119",
          "program" => "2867370d06d5",
          "message" => "172.16.0.226,48982,envoy,2023-03-08T03:42:39.340Z,34.142.199.10,443,-,www.envoyproxy.io,HTTP/1.1,GET,/,HTTP/1.1,200,17304,2023-03-08T03:42:39.340Z,17304,-,-,curl/7.29.0,-,1000,0,0,0,0,0,0,0,0,0,0,0",
    "syslog_abspri" => "30",
        "timestamp" => "Mar  8 11:42:45"
}

[Rios]

Or you can use the dissect filter, which I have completely forgotten.

  dissect {
        mapping => {
            "message" => "%{}]: %{message}"
        }
  }
# and csv filter with columns naming
  csv {
      separator => ","
      skip_header => "true"
      columns => ["source.ip","source.port","name","time","destination.ip","destination.port","method","host","HTTPver"]
  }