Patterns defined under patterns_dir are not valid

Elastic Stack Logstash
21

Hi @its-ogawa

So I just created these logs and followed this process and it worked for me.

NOTE it will not parse with your dummy data MY_SERVER_NAME etc

So I put in reasonable defaults

Feb  5 11:33:10 my.comp.domain postfix/smtp[2794930]: 1BF9DC0CC906: to=<test@gmail.com>, relay=gmail-smtp-in.l.google.com[192.168.0.1]:25, delay=1.2, delays=0.06/0.01/0.42/0.75, dsn=2.0.0, status=sent (250 2.0.0 OK  1707100390 l10-20020a170902e2ca00b001d741f0eefesi5356753plc.419 - gsmtp)
Feb  5 11:33:10 my.comp.domain postfix/smtp[2794930]: 1BF9DC0CC906: to=<test@gmail.com>, relay=gmail-smtp-in.l.google.com[192.168.0.1]:25, delay=1.2, delays=0.06/0.01/0.42/0.75, dsn=2.0.0, status=sent (250 2.0.0 OK  1707100390 l10-20020a170902e2ca00b001d741f0eefesi5356753plc.419 - gsmtp)

I added the patterns in my patterns directory, make sure this directory and file is readable

hyperion:logstash-patterns sbrown$ pwd
/Users/sbrown/workspace/sample-data/discuss/logstash-patterns
hyperion:logstash-patterns sbrown$ ls
pf-patterns
hyperion:logstash-patterns sbrown$

Here is my logstash.conf

input {
    file { path => "/Users/sbrown/workspace/sample-data/discuss/discuss-pf.log" sincedb_path => "/dev/null" start_position => beginning }
}

filter {
    grok {
        patterns_dir => ["/Users/sbrown/workspace/sample-data/discuss/logstash-patterns"]
        match => { "message" => "%{PF}" }
    }
}

output { stdout { } }

ran this command

sudo ./bin/logstash -r -f ./config/logstash-pf.conf

And here is my output..it works...

{
        "status" => "sent",
     "relayhost" => "gmail-smtp-in.l.google.com",
         "event" => {
        "original" => "Feb  5 11:33:10 my.comp.domain postfix/smtp[2794930]: 1BF9DC0CC906: to=<test@gmail.com>, relay=gmail-smtp-in.l.google.com[192.168.0.1]:25, delay=1.2, delays=0.06/0.01/0.42/0.75, dsn=2.0.0, status=sent (250 2.0.0 OK  1707100390 l10-20020a170902e2ca00b001d741f0eefesi5356753plc.419 - gsmtp)"
    },
     "relayport" => "25",
        "reason" => "(250 2.0.0 OK  1707100390 l10-20020a170902e2ca00b001d741f0eefesi5356753plc.419 - gsmtp)",
     "timestamp" => "Feb  5 11:33:10",
      "@version" => "1",
    "@timestamp" => 2024-02-06T16:02:06.960004Z,
       "message" => "Feb  5 11:33:10 my.comp.domain postfix/smtp[2794930]: 1BF9DC0CC906: to=<test@gmail.com>, relay=gmail-smtp-in.l.google.com[192.168.0.1]:25, delay=1.2, delays=0.06/0.01/0.42/0.75, dsn=2.0.0, status=sent (250 2.0.0 OK  1707100390 l10-20020a170902e2ca00b001d741f0eefesi5356753plc.419 - gsmtp)",
           "qid" => "1BF9DC0CC906",
       "relayip" => "192.168.0.1",
         "delay" => "1.2",
           "dsn" => "2.0.0",
     "logsource" => "my.comp.domain",
          "host" => {
        "name" => "hyperion"
    },
           "log" => {
        "file" => {
            "path" => "/Users/sbrown/workspace/sample-data/discuss/discuss-pf.log"
        }
    },
            "to" => "test@gmail.com",
       "process" => "smtp",
           "pid" => "2794930"
}

BTW

hyperion:logstash-patterns sbrown$ cat pf-patterns 
# Syslog stuff
PROCESS ([\w._\/%-]+)
COMPID postfix\/%{PROCESS:process}(?:\[%{NUMBER:pid}\])?
POSTFIX (?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{COMPID}:

# Postfix stuff
HELO (?:\[%{IP:helo}\]|%{HOSTNAME:helo}|%{DATA:helo})
QUEUEID (?:[A-F0-9]+|NOQUEUE)
EMAILADDRESSPART [a-zA-Z0-9_.+-=:~]+
EMAILADDRESS %{EMAILADDRESSPART:local}@%{EMAILADDRESSPART:remote}
RELAY (?:%{HOSTNAME:relayhost}(?:\[%{IP:relayip}\](?::%{BASE10NUM:relayport}?)?)?)
POSREAL [0-9]+(.[0-9]+)?
DSN %{NONNEGINT}.%{NONNEGINT}.%{NONNEGINT}
STATUS sent|deferred|bounced|expired
PERMERROR 5[0-9]{2}
MESSAGELEVEL reject|warning|error|fatal|panic

POSTFIXSMTPMESSAGE %{MESSAGELEVEL}: %{GREEDYDATA:reason}
POSTFIXACTION discard|dunno|filter|hold|ignore|info|prepend|redirect|replace|reject|warn

# postfix/smtp and postfix/lmtp, postfix/local and postfix/error
POSTFIXSMTP %{POSTFIXSMTPRELAY}|%{POSTFIXSMTPCONNECT}|%{POSTFIXSMTP5XX}|%{POSTFIXSMTPREFUSAL}|%{POSTFIXSMTPLOSTCONNECTION}|%{POSTFIXSMTPTIMEOUT}
POSTFIXSMTPRELAY %{QUEUEID:qid}: to=<%{DATA:to}>,(?:\sorig_to=<%{DATA:orig_to}>,)? relay=%{RELAY},(?: delay=%{POSREAL:delay},)?(?: delays=%{DATA:delays}?,)?(?: conn_use=%{POSREAL:conn_use},)?( %{WORD}=%{DATA},)+? dsn=%{DSN:dsn}, status=%{STATUS:status} (?:\(connect to %{POSTFIXSMTPCONNECTDEFERRED}|%{GREEDYDATA:reason})
POSTFIXSMTPCONNECT connect to %{RELAY}: %{GREEDYDATA:reason}
POSTFIXSMTPCONNECTDEFERRED %{RELAY}: %{GREEDYDATA:reason}\)
POSTFIXSMTP5XX %{QUEUEID:qid}: to=<%{EMAILADDRESS:to}>,(?:\sorig_to=<%{EMAILADDRESS:orig_to}>,)? relay=%{RELAY}, (%{WORD}=%{DATA},)+ dsn=%{DSN:dsn}, status=%{STATUS:status} \(host %{HOSTNAME}\[%{IP}\] said: %{PERMERROR:responsecode} %{DATA:smtp_response} \(in reply to %{DATA:command} command\)\)
POSTFIXSMTPREFUSAL %{QUEUEID:qid}: host %{RELAY} refused to talk to me: %{GREEDYDATA:reason}
POSTFIXSMTPLOSTCONNECTION %{QUEUEID:qid}: lost connection with %{RELAY} while %{GREEDYDATA:reason}
POSTFIXSMTPTIMEOUT %{QUEUEID:qid}: conversation with %{RELAY} timed out while %{GREEDYDATA:reason}

# postfix/smtpd
POSTFIXSMTPD %{POSTFIXSMTPDCONNECTS}|%{POSTFIXSMTPDACTIONS}|%{POSTFIXSMTPDTIMEOUTS}|%{POSTFIXSMTPDLOGIN}|%{POSTFIXSMTPDCLIENT}|%{POSTFIXSMTPDNOQUEUE}|%{POSTFIXSMTPDWARNING}|%{POSTFIXSMTPDLOSTCONNECTION}
POSTFIXSMTPDCONNECTS (?:dis)?connect from %{RELAY}
POSTFIXSMTPDACTIONS %{QUEUEID:qid}: %{POSTFIXACTION:postfix_action}: %{DATA:command} from %{RELAY}: %{PERMERROR:responsecode} %{DSN:dsn} %{DATA}: %{DATA:reason}; from=<%{EMAILADDRESS:from}> to=<%{EMAILADDRESS:to}> proto=%{DATA:proto} helo=<%{HELO}>
POSTFIXSMTPDTIMEOUTS timeout after %{DATA:command} from %{RELAY}
POSTFIXSMTPDLOGIN %{QUEUEID:qid}: client=%{DATA:client}, sasl_method=%{DATA:saslmethod}, sasl_username=%{GREEDYDATA:saslusername}
POSTFIXSMTPDCLIENT %{QUEUEID:qid}: client=%{GREEDYDATA:client}
POSTFIXSMTPDNOQUEUE NOQUEUE: %{POSTFIXACTION:postfix_action}: %{DATA:command} from %{RELAY}: %{GREEDYDATA:reason}
POSTFIXSMTPDWARNING warning:( %{IP}: | hostname %{HOSTNAME} )?%{GREEDYDATA:reason}
POSTFIXSMTPDLOSTCONNECTION (?:lost connection after %{DATA:smtp_response} from %{RELAY}|improper command pipelining after HELO from %{GREEDYDATA:reason})

# postfix/cleanup
POSTFIXCLEANUP %{POSTFIXCLEANUPMESSAGE}
POSTFIXCLEANUPMESSAGE %{QUEUEID:qid}: (resent-)?message-id=%{GREEDYDATA:messageid}

# postfix/bounce
POSTFIXBOUNCE %{QUEUEID:qid}: sender (non-)?delivery( status)? notification: %{QUEUEID:bouncequeueid}

# postfix/qmgr and postfix/pickup
POSTFIXQMGR %{QUEUEID:qid}: (?:removed|(?:uid=%{NUMBER:uid} )?from=<(?:%{DATA:from})?>(?:, size=%{NUMBER:size}, nrcpt=%{NUMBER:nrcpt} \(%{GREEDYDATA:queuestatus}\))?)

# postfix/anvil
POSTFIXANVIL statistics: %{GREEDYDATA:reason}

# postfix/trivial-rewrite
POSTFIXREWRITE warning: do not list domain %{DATA:domain} in BOTH mydestination and virtual_alias_domains

# Postfix wrap em up
PF %{POSTFIX} (?:%{POSTFIXSMTP}|%{POSTFIXANVIL}|%{POSTFIXQMGR}|%{POSTFIXBOUNCE}|%{POSTFIXCLEANUP}|%{POSTFIXSMTPD}|%{POSTFIXREWRITE})
1 Like
22

I sent to elasticsearch as well it worked fine

     {
        "_index": ".ds-logs-generic-default-2024.02.06-000001",
        "_id": "Ow86f40BnRySROIKb0jF",
        "_score": 1,
        "_source": {
          "reason": "(250 2.0.0 OK  1707100390 l10-20020a170902e2ca00b001d741f0eefesi5356753plc.419 - gsmtp)",
          "process": "smtp",
          "log": {
            "file": {
              "path": "/Users/sbrown/workspace/sample-data/discuss/discuss-pf.log"
            }
          },
          "relayport": "25",
          "relayip": "192.168.0.1",
          "relayhost": "gmail-smtp-in.l.google.com",
          "pid": "2794930",
          "logsource": "my.comp.domain",
          "message": "Feb  5 11:33:10 my.comp.domain postfix/smtp[2794930]: 1BF9DC0CC906: to=<test@gmail.com>, relay=gmail-smtp-in.l.google.com[192.168.0.1]:25, delay=1.2, delays=0.06/0.01/0.42/0.75, dsn=2.0.0, status=sent (250 2.0.0 OK  1707100390 l10-20020a170902e2ca00b001d741f0eefesi5356753plc.419 - gsmtp)",
          "qid": "1BF9DC0CC906",
          "@timestamp": "2024-02-06T16:22:32.279082Z",
          "delay": "1.2",
          "data_stream": {
            "namespace": "default",
            "type": "logs",
            "dataset": "generic"
          },
          "@version": "1",
          "host": {
            "name": "hyperion"
          },
          "to": "test@gmail.com",
          "event": {
            "original": "Feb  5 11:33:10 my.comp.domain postfix/smtp[2794930]: 1BF9DC0CC906: to=<test@gmail.com>, relay=gmail-smtp-in.l.google.com[192.168.0.1]:25, delay=1.2, delays=0.06/0.01/0.42/0.75, dsn=2.0.0, status=sent (250 2.0.0 OK  1707100390 l10-20020a170902e2ca00b001d741f0eefesi5356753plc.419 - gsmtp)"
          },
          "dsn": "2.0.0",
          "status": "sent",
          "timestamp": "Feb  5 11:33:10"
        }
      },
23

Thank you for going into detail.

I am sorry for the confusion I have caused where I have manually changed things for posting.
Also, thanks for rewriting it to the appropriate sample.

I have not taken any special steps.
but, I got the same results as you! (Miracle)

Your post will be helpful not only to me but to all Elastic lovers. Thank you so much.

1 Like
24

Can /tmp/patterns/foo be considered the same as the following postfix-grok-patterns?

Yes, if a user ("logstash") has rights to read the path.
Have you perhaps change the pattern path to any other after which was worked?

25

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.