Patterns defined under patterns_dir are not valid

Elastic Stack Logstash
21

Hi @its-ogawa

So I just created these logs and followed this process and it worked for me.

NOTE it will not parse with your dummy data MY_SERVER_NAME etc

So I put in reasonable defaults

Feb  5 11:33:10 my.comp.domain postfix/smtp[2794930]: 1BF9DC0CC906: to=<test@gmail.com>, relay=gmail-smtp-in.l.google.com[192.168.0.1]:25, delay=1.2, delays=0.06/0.01/0.42/0.75, dsn=2.0.0, status=sent (250 2.0.0 OK  1707100390 l10-20020a170902e2ca00b001d741f0eefesi5356753plc.419 - gsmtp)
Feb  5 11:33:10 my.comp.domain postfix/smtp[2794930]: 1BF9DC0CC906: to=<test@gmail.com>, relay=gmail-smtp-in.l.google.com[192.168.0.1]:25, delay=1.2, delays=0.06/0.01/0.42/0.75, dsn=2.0.0, status=sent (250 2.0.0 OK  1707100390 l10-20020a170902e2ca00b001d741f0eefesi5356753plc.419 - gsmtp)

I added the patterns in my patterns directory, make sure this directory and file is readable

hyperion:logstash-patterns sbrown$ pwd
/Users/sbrown/workspace/sample-data/discuss/logstash-patterns
hyperion:logstash-patterns sbrown$ ls
pf-patterns
hyperion:logstash-patterns sbrown$

Here is my logstash.conf

input {
    file { path => "/Users/sbrown/workspace/sample-data/discuss/discuss-pf.log" sincedb_path => "/dev/null" start_position => beginning }
}

filter {
    grok {
        patterns_dir => ["/Users/sbrown/workspace/sample-data/discuss/logstash-patterns"]
        match => { "message" => "%{PF}" }
    }
}

output { stdout { } }

ran this command

sudo ./bin/logstash -r -f ./config/logstash-pf.conf

And here is my output..it works...

{
        "status" => "sent",
     "relayhost" => "gmail-smtp-in.l.google.com",
         "event" => {
        "original" => "Feb  5 11:33:10 my.comp.domain postfix/smtp[2794930]: 1BF9DC0CC906: to=<test@gmail.com>, relay=gmail-smtp-in.l.google.com[192.168.0.1]:25, delay=1.2, delays=0.06/0.01/0.42/0.75, dsn=2.0.0, status=sent (250 2.0.0 OK  1707100390 l10-20020a170902e2ca00b001d741f0eefesi5356753plc.419 - gsmtp)"
    },
     "relayport" => "25",
        "reason" => "(250 2.0.0 OK  1707100390 l10-20020a170902e2ca00b001d741f0eefesi5356753plc.419 - gsmtp)",
     "timestamp" => "Feb  5 11:33:10",
      "@version" => "1",
    "@timestamp" => 2024-02-06T16:02:06.960004Z,
       "message" => "Feb  5 11:33:10 my.comp.domain postfix/smtp[2794930]: 1BF9DC0CC906: to=<test@gmail.com>, relay=gmail-smtp-in.l.google.com[192.168.0.1]:25, delay=1.2, delays=0.06/0.01/0.42/0.75, dsn=2.0.0, status=sent (250 2.0.0 OK  1707100390 l10-20020a170902e2ca00b001d741f0eefesi5356753plc.419 - gsmtp)",
           "qid" => "1BF9DC0CC906",
       "relayip" => "192.168.0.1",
         "delay" => "1.2",
           "dsn" => "2.0.0",
     "logsource" => "my.comp.domain",
          "host" => {
        "name" => "hyperion"
    },
           "log" => {
        "file" => {
            "path" => "/Users/sbrown/workspace/sample-data/discuss/discuss-pf.log"
        }
    },
            "to" => "test@gmail.com",
       "process" => "smtp",
           "pid" => "2794930"
}

BTW

hyperion:logstash-patterns sbrown$ cat pf-patterns 
# Syslog stuff
PROCESS ([\w._\/%-]+)
COMPID postfix\/%{PROCESS:process}(?:\[%{NUMBER:pid}\])?
POSTFIX (?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{COMPID}:

# Postfix stuff
HELO (?:\[%{IP:helo}\]|%{HOSTNAME:helo}|%{DATA:helo})
QUEUEID (?:[A-F0-9]+|NOQUEUE)
EMAILADDRESSPART [a-zA-Z0-9_.+-=:~]+
EMAILADDRESS %{EMAILADDRESSPART:local}@%{EMAILADDRESSPART:remote}
RELAY (?:%{HOSTNAME:relayhost}(?:\[%{IP:relayip}\](?::%{BASE10NUM:relayport}?)?)?)
POSREAL [0-9]+(.[0-9]+)?
DSN %{NONNEGINT}.%{NONNEGINT}.%{NONNEGINT}
STATUS sent|deferred|bounced|expired
PERMERROR 5[0-9]{2}
MESSAGELEVEL reject|warning|error|fatal|panic

POSTFIXSMTPMESSAGE %{MESSAGELEVEL}: %{GREEDYDATA:reason}
POSTFIXACTION discard|dunno|filter|hold|ignore|info|prepend|redirect|replace|reject|warn

# postfix/smtp and postfix/lmtp, postfix/local and postfix/error
POSTFIXSMTP %{POSTFIXSMTPRELAY}|%{POSTFIXSMTPCONNECT}|%{POSTFIXSMTP5XX}|%{POSTFIXSMTPREFUSAL}|%{POSTFIXSMTPLOSTCONNECTION}|%{POSTFIXSMTPTIMEOUT}
POSTFIXSMTPRELAY %{QUEUEID:qid}: to=<%{DATA:to}>,(?:\sorig_to=<%{DATA:orig_to}>,)? relay=%{RELAY},(?: delay=%{POSREAL:delay},)?(?: delays=%{DATA:delays}?,)?(?: conn_use=%{POSREAL:conn_use},)?( %{WORD}=%{DATA},)+? dsn=%{DSN:dsn}, status=%{STATUS:status} (?:\(connect to %{POSTFIXSMTPCONNECTDEFERRED}|%{GREEDYDATA:reason})
POSTFIXSMTPCONNECT connect to %{RELAY}: %{GREEDYDATA:reason}
POSTFIXSMTPCONNECTDEFERRED %{RELAY}: %{GREEDYDATA:reason}\)
POSTFIXSMTP5XX %{QUEUEID:qid}: to=<%{EMAILADDRESS:to}>,(?:\sorig_to=<%{EMAILADDRESS:orig_to}>,)? relay=%{RELAY}, (%{WORD}=%{DATA},)+ dsn=%{DSN:dsn}, status=%{STATUS:status} \(host %{HOSTNAME}\[%{IP}\] said: %{PERMERROR:responsecode} %{DATA:smtp_response} \(in reply to %{DATA:command} command\)\)
POSTFIXSMTPREFUSAL %{QUEUEID:qid}: host %{RELAY} refused to talk to me: %{GREEDYDATA:reason}
POSTFIXSMTPLOSTCONNECTION %{QUEUEID:qid}: lost connection with %{RELAY} while %{GREEDYDATA:reason}
POSTFIXSMTPTIMEOUT %{QUEUEID:qid}: conversation with %{RELAY} timed out while %{GREEDYDATA:reason}

# postfix/smtpd
POSTFIXSMTPD %{POSTFIXSMTPDCONNECTS}|%{POSTFIXSMTPDACTIONS}|%{POSTFIXSMTPDTIMEOUTS}|%{POSTFIXSMTPDLOGIN}|%{POSTFIXSMTPDCLIENT}|%{POSTFIXSMTPDNOQUEUE}|%{POSTFIXSMTPDWARNING}|%{POSTFIXSMTPDLOSTCONNECTION}
POSTFIXSMTPDCONNECTS (?:dis)?connect from %{RELAY}
POSTFIXSMTPDACTIONS %{QUEUEID:qid}: %{POSTFIXACTION:postfix_action}: %{DATA:command} from %{RELAY}: %{PERMERROR:responsecode} %{DSN:dsn} %{DATA}: %{DATA:reason}; from=<%{EMAILADDRESS:from}> to=<%{EMAILADDRESS:to}> proto=%{DATA:proto} helo=<%{HELO}>
POSTFIXSMTPDTIMEOUTS timeout after %{DATA:command} from %{RELAY}
POSTFIXSMTPDLOGIN %{QUEUEID:qid}: client=%{DATA:client}, sasl_method=%{DATA:saslmethod}, sasl_username=%{GREEDYDATA:saslusername}
POSTFIXSMTPDCLIENT %{QUEUEID:qid}: client=%{GREEDYDATA:client}
POSTFIXSMTPDNOQUEUE NOQUEUE: %{POSTFIXACTION:postfix_action}: %{DATA:command} from %{RELAY}: %{GREEDYDATA:reason}
POSTFIXSMTPDWARNING warning:( %{IP}: | hostname %{HOSTNAME} )?%{GREEDYDATA:reason}
POSTFIXSMTPDLOSTCONNECTION (?:lost connection after %{DATA:smtp_response} from %{RELAY}|improper command pipelining after HELO from %{GREEDYDATA:reason})

# postfix/cleanup
POSTFIXCLEANUP %{POSTFIXCLEANUPMESSAGE}
POSTFIXCLEANUPMESSAGE %{QUEUEID:qid}: (resent-)?message-id=%{GREEDYDATA:messageid}

# postfix/bounce
POSTFIXBOUNCE %{QUEUEID:qid}: sender (non-)?delivery( status)? notification: %{QUEUEID:bouncequeueid}

# postfix/qmgr and postfix/pickup
POSTFIXQMGR %{QUEUEID:qid}: (?:removed|(?:uid=%{NUMBER:uid} )?from=<(?:%{DATA:from})?>(?:, size=%{NUMBER:size}, nrcpt=%{NUMBER:nrcpt} \(%{GREEDYDATA:queuestatus}\))?)

# postfix/anvil
POSTFIXANVIL statistics: %{GREEDYDATA:reason}

# postfix/trivial-rewrite
POSTFIXREWRITE warning: do not list domain %{DATA:domain} in BOTH mydestination and virtual_alias_domains

# Postfix wrap em up
PF %{POSTFIX} (?:%{POSTFIXSMTP}|%{POSTFIXANVIL}|%{POSTFIXQMGR}|%{POSTFIXBOUNCE}|%{POSTFIXCLEANUP}|%{POSTFIXSMTPD}|%{POSTFIXREWRITE})
22

I sent to elasticsearch as well it worked fine

     {
        "_index": ".ds-logs-generic-default-2024.02.06-000001",
        "_id": "Ow86f40BnRySROIKb0jF",
        "_score": 1,
        "_source": {
          "reason": "(250 2.0.0 OK  1707100390 l10-20020a170902e2ca00b001d741f0eefesi5356753plc.419 - gsmtp)",
          "process": "smtp",
          "log": {
            "file": {
              "path": "/Users/sbrown/workspace/sample-data/discuss/discuss-pf.log"
            }
          },
          "relayport": "25",
          "relayip": "192.168.0.1",
          "relayhost": "gmail-smtp-in.l.google.com",
          "pid": "2794930",
          "logsource": "my.comp.domain",
          "message": "Feb  5 11:33:10 my.comp.domain postfix/smtp[2794930]: 1BF9DC0CC906: to=<test@gmail.com>, relay=gmail-smtp-in.l.google.com[192.168.0.1]:25, delay=1.2, delays=0.06/0.01/0.42/0.75, dsn=2.0.0, status=sent (250 2.0.0 OK  1707100390 l10-20020a170902e2ca00b001d741f0eefesi5356753plc.419 - gsmtp)",
          "qid": "1BF9DC0CC906",
          "@timestamp": "2024-02-06T16:22:32.279082Z",
          "delay": "1.2",
          "data_stream": {
            "namespace": "default",
            "type": "logs",
            "dataset": "generic"
          },
          "@version": "1",
          "host": {
            "name": "hyperion"
          },
          "to": "test@gmail.com",
          "event": {
            "original": "Feb  5 11:33:10 my.comp.domain postfix/smtp[2794930]: 1BF9DC0CC906: to=<test@gmail.com>, relay=gmail-smtp-in.l.google.com[192.168.0.1]:25, delay=1.2, delays=0.06/0.01/0.42/0.75, dsn=2.0.0, status=sent (250 2.0.0 OK  1707100390 l10-20020a170902e2ca00b001d741f0eefesi5356753plc.419 - gsmtp)"
          },
          "dsn": "2.0.0",
          "status": "sent",
          "timestamp": "Feb  5 11:33:10"
        }
      },
23

Thank you for going into detail.

I am sorry for the confusion I have caused where I have manually changed things for posting.
Also, thanks for rewriting it to the appropriate sample.

I have not taken any special steps.
but, I got the same results as you! (Miracle)

Your post will be helpful not only to me but to all Elastic lovers. Thank you so much.

24

Can /tmp/patterns/foo be considered the same as the following postfix-grok-patterns?

Yes, if a user ("logstash") has rights to read the path.
Have you perhaps change the pattern path to any other after which was worked?