Hi and thanks for reading.
I'm trying to do my own postfix/imapd logs filter.
This is what i made:
%{SYSLOGTIMESTAMP:timestamp} %{WORD:mail} %{WORD:program}/?(%{WORD:subprogram})?\['?(%{BASE10NUM:id})?\]?:? %{GREEDYDATA:message}
And this is my grok filter
filter {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{WORD:mail} %{WORD:program}/?(%{WORD:subprogram})?\[?(%{BASE10NUM:id})?\]?:? %{GREEDYDATA:message}" }
}
At grokdebug.herokuapp.com it works. But Logstash give me this:
{:timestamp=>"2016-09-19T13:18:07.660000-0300", :message=>"fetched an invalid config", :config=>"input {\r\n\tfile {\r\n\t\tpath => \"/var/log/maillog\"\r\n\t\ttype => \"maillog\"\r\n\t\tstart_position => \"beginning\"\r\n\t\tignore_older => 0\r\n\t}\r\n}\r\n\r\nfilter {\r\n\tgrok {\r\n\t\tmatch => { \"message\" => \"%{SYSLOGTIMESTAMP:timestamp} %{WORD:mail} %{WORD:program}/?(%{WORD:subprogram})?\\[?(%{BASE10NUM:id})?\\]?:? %{GREEDYDATA:message}\" }\r\n}\r\n\r\noutput {\r\n\telasticsearch {\r\n\t\thosts => [ \"localhost:9200\" ]\r\n\t}\r\n}\r\n\n", :reason=>"Expected one of #, => at line 16, column 16 (byte 342) after filter {\r\n\tgrok {\r\n\t\tmatch => { \"message\" => \"%{SYSLOGTIMESTAMP:timestamp} %{WORD:mail} %{WORD:program}/?(%{WORD:subprogram})?\\[?(%{BASE10NUM:id})?\\]?:? %{GREEDYDATA:message}\" }\r\n}\r\n\r\noutput {\r\n\telasticsearch ", :level=>:error}
Hope this helps to understand. Thanks!
It looks like there's a closing brace missing; you're never closing the filter block.
1 Like
Thanks for reply. I fixed this problem just doing what you said.
I keep going on with this. Now i have another issue (i don't want to open a new topic).
I have this filter:
> filter {
> grok {
> match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{WORD:mail} %{WORD:program}/?(%{WORD:subprogram})?[?(%{BASE10NUM:id})?]?:?} %{WORD:action}%{GREEDYDATA:action_info}" }
> }
> if [action] == "disconnect" or [action] == "connect" {
> grok {
> match => { "action_info" => " from %{GREEDYDATA:host}[%{IPV4:ip_host}]" }
> }
> }
> }
And it is not working. Logstash add the tag "_grokparsefailure" for each log.
..."tags":["_grokparsefailure"]}
Please show the whole event as produced by a stdout { codec => rubydebug }
output.
The stdout { codec => rubydebug }
output is too long to put here, but is extremely repetitive, so i copy a paragraph:
{
"message" => "Sep 18 04:40:15 mail postfix/smtpd[1979]: connect from localhost[127.0.0.1]",
"@version" => "1",
"@timestamp" => "2016-09-19T20:15:53.638Z",
"path" => "/var/log/dsaasd",
"host" => "ELK-System",
"type" => "registro_mail",
"tags" => [
[0] "_grokparsefailure"
]
}
Multiple problems, mainly here: [?(%{BASE10NUM:id})?]?:?}
- The square brackets must be escaped.
- Why all the question marks everywhere?
- What's up with the closing brace at the end?
Suggestion if the bracketed pid should be optional: (\[%{BASE10NUM:id}\])?
When debugging things like this, always start with a short expression, in this case e.g. %{SYSLOGTIMESTAMP:timestamp}
, then add more and more until things break.