My grok fikter dont work

Elastic Stack Logstash
1

Hi, I want the logs of my service to be processed using the filter grok. But instead I get _grokparsefailure.
Could you tell me what the problem is?

My logstash configuration:

input {
        file {
                path => ["/path/to/log/*.log"]
                exclude => [ "*.gz", "*.zip", "*.rar","*.xlog"]
                start_position => "end"
                discover_interval => 1
        }
}
filter {
        grok{
                match => {"message" => "%{INT:count}[T ]%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day}[T ]%{HOUR:hour}:?%{MINUTE:min}(?::?%{SECOND:sec})?[T ]\(?%{BASE10NUM}\:?%{BASE16NUM}\)[T ]\[?%{LOGLEVEL:exception_level}\s*][T ]\:[T ]\<?%{USERNAME:user}\>[T ](%{GREEDYDATA:message})?"}
        }
}
output {
        elasticsearch {
                hosts => "http://IP_ADDR:9200"
                index => "logs-%{+YYYY.MM.dd}"
        }
}

My logs example:

1 2018-05-04 12:02:47.852003 (3476:0x00007f466d7fb700) [WARNING] : <Root> Failed to close session to service

My index:
{
"_index": "logs-2018.06.28",
"_type": "logs",
"_id": "AWRGVgCw5z-x38VXIohu",
"_score": 1,
"_source": {
"path": "/path/to/log/***.log",
"@timestamp": "2018-06-28T12:19:58.752Z",
"@version": "1",
"host": "RHEL6.local",
"message": "1 2018-05-04 12:02:47.852003 (3476:0x00007f466d7fb700) [WARNING] : Failed to close session to service ",
"tags": [
"_grokparsefailure"
]
}
}

2

I can't spot anything off the top of my head, but just build the expression gradually and you'll find the problem. Start with ^%{INT:count}[T ] and build from there.

3

I tried this method, nothing has changed.
But I did not try to put the symbol "^" at the beginning

4

Hello!
I try your method, I dont see anymore "_grokparsefailure" tag.
But, my message dont cut

> {
>         "_index": "logs-2018.07.04",
>         "_type": "logs",
>         "_id": "AWRkABAAMgfICQ_NuSAY",
>         "_score": 1,
>         "_source": {
>           "path": "/path/to/log/***.log",
>           "@timestamp": "2018-07-04T06:34:43.056Z",
>           "@version": "1",
>           "host": "RHEL6.local",
>           "count": "2",
>           "message": "2 2018-07-04 09:34:42.980795 (52245:0x00007f1087fff700) [WARNING] : <Iscp> Failed to connect to sniffer. Error: connect: Connection refused"
>         }

Maybe, I make some mistake, because "count" field is determined.

New grok:

> ^%{INT:count}[T ]^%{YEAR:year}-^%{MONTHNUM:month}-^%{MONTHDAY:day}[T ]^%{HOUR:hour}:?^%{MINUTE:min}(?::?^%{SECOND:sec})?[T ]\(?^%{BASE10NUM}\:?^%{BASE16NUM}\)[T ]\[?^%{LOGLEVEL:exception_level}\s*][T ]\:[T ]\<?^%{USERNAME:user}\>[T ](^%{GREEDYDATA:message})?

5

Remove all ^ except the first one. That character means "beginning of the line", which obviously only should occur prior to %{INT:count}:

6

Thank you so much!
It works!
If I still have a question, should I open a new topic, or continue communication in this?

7

Unless it's directly related to the existing topic please start a new one.