Hi, I want the logs of my service to be processed using the filter grok. But instead I get _grokparsefailure.
Could you tell me what the problem is?
My logstash configuration:
input {
file {
path => ["/path/to/log/*.log"]
exclude => [ "*.gz", "*.zip", "*.rar","*.xlog"]
start_position => "end"
discover_interval => 1
}
}
filter {
grok{
match => {"message" => "%{INT:count}[T ]%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day}[T ]%{HOUR:hour}:?%{MINUTE:min}(?::?%{SECOND:sec})?[T ]\(?%{BASE10NUM}\:?%{BASE16NUM}\)[T ]\[?%{LOGLEVEL:exception_level}\s*][T ]\:[T ]\<?%{USERNAME:user}\>[T ](%{GREEDYDATA:message})?"}
}
}
output {
elasticsearch {
hosts => "http://IP_ADDR:9200"
index => "logs-%{+YYYY.MM.dd}"
}
}
My logs example:
1 2018-05-04 12:02:47.852003 (3476:0x00007f466d7fb700) [WARNING] : <Root> Failed to close session to service
My index:
{
"_index": "logs-2018.06.28",
"_type": "logs",
"_id": "AWRGVgCw5z-x38VXIohu",
"_score": 1,
"_source": {
"path": "/path/to/log/***.log",
"@timestamp ": "2018-06-28T12:19:58.752Z",
"@version ": "1",
"host": "RHEL6.local",
"message": "1 2018-05-04 12:02:47.852003 (3476:0x00007f466d7fb700) [WARNING] : Failed to close session to service ",
"tags": [
"_grokparsefailure"
]
}
}
I can't spot anything off the top of my head, but just build the expression gradually and you'll find the problem. Start with ^%{INT:count}[T ]
and build from there.
1 Like
I tried this method, nothing has changed.
But I did not try to put the symbol "^" at the beginning
Hello!
I try your method, I dont see anymore "_grokparsefailure" tag.
But, my message dont cut
> {
> "_index": "logs-2018.07.04",
> "_type": "logs",
> "_id": "AWRkABAAMgfICQ_NuSAY",
> "_score": 1,
> "_source": {
> "path": "/path/to/log/***.log",
> "@timestamp": "2018-07-04T06:34:43.056Z",
> "@version": "1",
> "host": "RHEL6.local",
> "count": "2",
> "message": "2 2018-07-04 09:34:42.980795 (52245:0x00007f1087fff700) [WARNING] : <Iscp> Failed to connect to sniffer. Error: connect: Connection refused"
> }
Maybe, I make some mistake, because "count" field is determined.
New grok:
> ^%{INT:count}[T ]^%{YEAR:year}-^%{MONTHNUM:month}-^%{MONTHDAY:day}[T ]^%{HOUR:hour}:?^%{MINUTE:min}(?::?^%{SECOND:sec})?[T ]\(?^%{BASE10NUM}\:?^%{BASE16NUM}\)[T ]\[?^%{LOGLEVEL:exception_level}\s*][T ]\:[T ]\<?^%{USERNAME:user}\>[T ](^%{GREEDYDATA:message})?
Remove all ^
except the first one. That character means "beginning of the line", which obviously only should occur prior to %{INT:count}
:
Thank you so much!
It works!
If I still have a question, should I open a new topic, or continue communication in this?
Unless it's directly related to the existing topic please start a new one.
system
(system)
Closed
August 10, 2018, 11:35am
8
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.