Grok can't match my log

Elastic Stack Logstash
1

My log:

[23:59:52.743]2018-08-14 23:59:52 DEBUG [resin-tcp-connection-*:8012-59] /queryup[sversion=5][model=HUAWEI G7-UL20][from=_360][lastupdatetime=1532506757000][versioncode=201805140][userId=13994251][mac=863846027484619][version=5.2.7][uniqid=974846027484619][needsecretary=true][os=Android4.4.4][screen=720,1184][token=8737a7830aa0c5a14633abd300187af1][appid=1][agent=testBizCard/5.2.7][ip=113.231.11.12][session=abcsZQsuoJfY-6xkJk5uw] [com.test.mobile.interceptor.LoggerInterceptor.before(63)]

logstash config:

input {
  file {
      path => [ "/data/test/*" ]
      start_position => "beginning"
      tags =>  "test"
      type => "test"
      codec=> multiline {
         pattern => "^\[([0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3})\]([0-9]{4}-[0-9]{2}-[0-9]{2})"
         negate => true
         what => "previous"
      }
  }
}

filter {
  if [type] == "test" {

      grok {
            patterns_dir => "/etc/logstash/mypatterns"
            match => [

                "message" , "\[%{TIME:time}\](20%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day} %{HOUR}:%{MINUTE}:%{SECOND}) %{LOGLEVEL:level} %{NOTSPACE} %{URIPATH:uripath}%{DATA}%{MODEL:model}%{DATA}%{USERID:userId}%{DATA}%{VERSION:version}%{DATA}}%{CLIENTOS:clientos}"

            ]
      }

      mutate {
           add_field => [ "test_time","20%{year}-%{month}-%{day} %{time}"]
      }


      date {
            match => [ "test_time","yyyy-MM-dd HH:mm:ss.SSS" ]
      }

      mutate {
           remove_field => ["test_time","year","month","day","time"]
      }

  }
}

output {

  if [type] == "test" {
    file {
        path => "/data/debug.log"

    }
    stdout{
        codec => rubydebug
    }
  }

}

My custom grok pattern: /etc/logstash/mypatterns

MODEL model=(.*?)(?=])
USERID userId=(.*?)(?=])
VERSION version=(.*?)(?=])
CLIENTOS os=(.*?)(?=])
CLIENTIP ip=(.*?)(?=])

I'm testing my grok pattern successful in 'http://grokdebug.herokuapp.com/',but Running in logstash like above configuration does't mach my log.

logstash erro log:

{"path":"/data/test/mobile/stdout.log.3","tags":["test","_grokparsefailure","_dateparsefailure"],"type":"test","message":"[23:59:52.743]2018-08-14 23:59:52 DEBUG [resin-tcp-connection-*:8012-59] /queryup[sversion=5][model=HUAWEI G7-UL20][from=_360][lastupdatetime=1532506757000][versioncode=201805140][userId=13994251][mac=863846027484619][version=5.2.7][uniqid=974846027484619][needsecretary=true][os=Android4.4.4][screen=720,1184][token=8737a7830aa0c5a14633abd300187af1][appid=1][agent=testBizCard/5.2.7][ip=113.231.11.12][session=abcsZQsuoJfY-6xkJk5uw] [com.test.mobile.interceptor.LoggerInterceptor.before(63)]","@timestamp":"2018-09-25T02:45:36.011Z","@version":"1","host":"rkeserver.bcc.global"}

If I delete the last two parts, it can match my log,the configurgration like :

     grok {
            patterns_dir => "/etc/logstash/mypatterns"
            match => [

                "message" , "\[%{TIME:time}\](20%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day} %{HOUR}:%{MINUTE}:%{SECOND}) %{LOGLEVEL:level} %{NOTSPACE} %{URIPATH:uripath}%{DATA}%{MODEL:model}%{DATA}%{USERID:userId}%{DATA}%{VERSION:version}
            ]
      }
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.