Hi All,
I am new on this forum, and a new user of ELK. I will try to describe my problem.
I'm trying to parse Zimbra log file and I begin with these two lines :
Aug 4 06:44:30 ags-zimbraprox postfix/smtpd[7440]: D3F8F630CF: client=ags-mailcube.ags-hosting.fr[185.50.52.77]
Aug 4 06:44:30 ags-zimbraprox postfix/cleanup[7371]: D3F8F630CF: message-id=CAGszSQVZyju-SugnvKmzVSA1L0D6AkOcc+VL4qL6X_OksGL47g@mail.gmail.com
I have create my logstash conf file like this :
input { stdin {} }
filter {
grok {
patterns_dir => ["/etc/logstash/patterns"]
match => {"message" => "%{POSTFIXCLEANUP}"}
}
if "_grokparsefailure" in [tags] {
grok {
patterns_dir => ["/etc/logstash/patterns"]
match => {"message" => "%{REMOTESMTP}"}
}
}
}
output {
elasticsearch { hosts => ["10.34.0.36:9200"]
index => "zimbra3"
#document_type => "my_index_zimbra"
}
stdout { codec => rubydebug }
}
And create some patterns :
QUEUEID (?:[a-zA-Z0-9]+|NOQUEUE)
POSTFIXPROTOCOL smtpd|cleanup
POSTFIXREQUEST ags-zimbraprox postfix/%{POSTFIXPROTOCOL}[[0-9]*]
DATENT [a-zA-Z]+ +[0-9]{1,2} +[0-9]{2}:[0-9]{2}:[0-9]{2}
POSTFIXCLEANUP %{DATENT:datejour} %{POSTFIXREQUEST:preq}: %{QUEUEID:qid}: message-id=<%{DATA:messageid}>
REMOTESMTP %{DATENT:datejour} %{POSTFIXREQUEST:preq}: %{QUEUEID:qid}: client=%{HOSTNAME:clienthost}[%{IP:clientip}]
When I execute the following command to test my conf : head -50 /home/admin-ags/zimbra.log | /opt/logstash/bin/logstash -f conf.d/zimbra_log.conf
It returns :
Settings: Default pipeline workers: 1
Pipeline main started
{
"message" => "Aug 4 06:44:30 ags-zimbraprox postfix/smtpd[7440]: D3F8F630CF: client=ags-mailcube.ags-hosting.fr[185.50.52.77]\r",
"@version" => "1",
"@timestamp" => "2017-08-04T16:38:53.762Z",
"host" => "AGS-ELK",
"tags" => [
[0] "_grokparsefailure"
],
"datejour" => "Aug 4 06:44:30",
"preq" => "ags-zimbraprox postfix/smtpd[7440]",
"qid" => "D3F8F630CF",
"clienthost" => "ags-mailcube.ags-hosting.fr",
"clientip" => "185.50.52.77"
}
{
"message" => "Aug 4 06:44:30 ags-zimbraprox postfix/cleanup[7371]: D3F8F630CF: message-id=CAGszSQVZyju-SugnvKmzVSA1L0D6AkOcc+VL4qL6X_OksGL47g@mail.gmail.com\r",
"@version" => "1",
"@timestamp" => "2017-08-04T16:38:55.079Z",
"host" => "AGS-ELK",
"qid" => "D3F8F630CF",
"messageid" => "CAGszSQVZyju-SugnvKmzVSA1L0D6AkOcc+VL4qL6X_OksGL47g@mail.gmail.com>\r"
}
Pipeline main has been shutdown
stopping pipeline {:id=>"main"}
And I don't understand why for my second line, it doesn't return the datejour and preq value.
Have you got any idea ?
Thanks for your help.