Patterns defined under patterns_dir are not valid

its-ogawa · February 5, 2024, 3:25am

I would like to write a grok pattern for logstash using patterns_dir for maillog based on the following document.

There is a postfix-grok-patterns file in patterns_dir with all the following patterns.

padusumilli/postfix-grok/blob/master/postfix-grok-patterns

# Syslog stuff
PROCESS ([\w._\/%-]+)
COMPID postfix\/%{PROCESS:process}(?:\[%{NUMBER:pid}\])?
POSTFIX (?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{COMPID}:

# Postfix stuff
HELO (?:\[%{IP:helo}\]|%{HOSTNAME:helo}|%{DATA:helo})
QUEUEID (?:[A-F0-9]+|NOQUEUE)
EMAILADDRESSPART [a-zA-Z0-9_.+-=:~]+
EMAILADDRESS %{EMAILADDRESSPART:local}@%{EMAILADDRESSPART:remote}
RELAY (?:%{HOSTNAME:relayhost}(?:\[%{IP:relayip}\](?::%{BASE10NUM:relayport}?)?)?)
POSREAL [0-9]+(.[0-9]+)?
DSN %{NONNEGINT}.%{NONNEGINT}.%{NONNEGINT}
STATUS sent|deferred|bounced|expired
PERMERROR 5[0-9]{2}
MESSAGELEVEL reject|warning|error|fatal|panic

POSTFIXSMTPMESSAGE %{MESSAGELEVEL}: %{GREEDYDATA:reason}
POSTFIXACTION discard|dunno|filter|hold|ignore|info|prepend|redirect|replace|reject|warn

This file has been truncated. show original

I have added the following patterns_dir to my logstash.

# view /etc/logstash/logstash.conf

input {
    beats {
        port => "5044"
    }
}
filter {
  ... snip ...
  ### For Mail log --->
  if( [tag][1] == "mail" or "mail" in [log][file][path] ) {
    grok {
       patterns_dir => ["/etc/logstash/patterns"]  # <- add
       match => { "message" => "{%PF}" }           # <- add
    }
  }
  ### <--- For Mail log
  ... snip ...
}
output {
  elasticsearch {
     hosts => ["localhost"]
     index => "%{[fields][index_name]}-%{+YYYY.MM.dd}"
     #ilm_enabled => true
  }
}

However, when I check with Kibana's Discover, the name defined in postfix-grok-patterns does not appear in the field.
I wonder why?

What I mean, specifically, is to get the to email address and delay from the following log, but neither the to nor the delay appear in the fields on Kibana, but all the contents appear in the message.

Feb  5 11:33:10 MY_SERVER_NAME postfix/smtp[2794930]: 1BF9DC0CC906: to=<XXXXXX@gmail.com>, relay=gmail-smtp-in.l.google.com[XXX.XXX.XXX.XXX]:25, delay=1.2, delays=0.06/0.01/0.42/0.75, dsn=2.0.0, status=sent (250 2.0.0 OK  1707100390 l10-20020a170902e2ca00b001d741f0eefesi5356753plc.419 - gsmtp)

The Grok Debbuger within Kibana (and the contents of the postfix-grok-patterns file) yields the following results.

{
  "reason": "(250 2.0.0 OK  1707100390 l10-20020a170902e2ca00b001d741f0eefesi5356753plc.419 - gsmtp)",
  "process": "smtp",
  "relayport": "25",
  "relayip": "XXX.XXX.XXX.XXX",
  "relayhost": "gmail-smtp-in.l.google.com",
  "pid": "2794930",
  "logsource": "MY_SERVER_NAME",
  "qid": "1BF9DC0CC906",
  "delay": "1.2",
  "to": "XXXX@gmail.com",
  "dsn": "2.0.0",
  "timestamp": "Feb  5 11:33:10",
  "status": "sent"
}

How can I see the above results as fields in Kibana's Discover?

Badger · February 5, 2024, 3:49am

Can you show us an example of one of your maillog messages and the resulting event?

its-ogawa · February 5, 2024, 4:09am

Thanks for the reply.
Here is an example of one of the mail log messages and the resulting events

log:

Feb  5 12:00:34 MY_SERVER_NAME postfix/smtp[3130871]: 5C3E5C0A4CDC: to=<XXXXX@yahoo.co.jp>, relay=mx5.mail.yahoo.co.jp[XXX.XXX.XXX.XXX]:25, conn_use=4, delay=3.4, delays=0.05/0/0.06/3.3, dsn=5.0.0, status=bounced (host mx5.mail.yahoo.co.jp[XXX.XXX.XXX.XXX] said: 554 delivery error: dd This user doesn't have a yahoo.co.jp account (XXXXX@yahoo.co.jp) [-5] - mta7143.mail.djm.ynwp.yahoo.co.jp (in reply to end of DATA command))

result:

{
  "_index": "MY_SERVER_NAME-mail-2024.02.05",
  "_id": "xj01d40Btc7-JZV64zlo",
  "_version": 1,
  "_score": 0,
  "_ignored": [
    "message.keyword",
    "event.original.keyword"
  ],
  "_source": {
    "agent": {
      "version": "7.12.1",
      "name": "MY_SERVER_NAME",
      "id": "b6e3e704-1ef9-4bfe-980d-973b6141e02b",
      "hostname": "MY_SERVER_NAME",
      "type": "filebeat",
      "ephemeral_id": "56093b18-ea05-4d11-b240-151139f9af9c"
    },
    "tags": [
      "kin",
      "mail",
      "beats_input_codec_plain_applied",
      "_grokparsefailure"
    ],
    "log": {
      "file": {
        "path": "/var/log/maillog"
      },
      "offset": 1506901
    },
    "input": {
      "type": "log"
    },
    "host": {
      "containerized": false,
      "os": {
        "version": "8.6 (Green Obsidian)",
        "name": "Rocky Linux",
        "kernel": "4.18.0-372.9.1.el8.x86_64",
        "family": "",
        "platform": "rocky",
        "type": "linux"
      },
      "name": "MY_SERVER_NAME",
      "mac": [
        "00:50:56:89:5a:dc",
        "00:50:56:89:26:34"
      ],
      "ip": [
        "XXX.XXX.XXX.XXX",
        "XXX.XXX.XXX.XXX"
      ],
      "id": "b4a712b2cebb41268aa683d962977dde",
      "hostname": "MY_SERVER_NAME",
      "architecture": "x86_64"
    },
    "message": "Feb  5 12:00:34 MY_SERVER_NAME postfix/smtp[3130871]: 5C3E5C0A4CDC: to=<XXXXX@yahoo.co.jp>, relay=mx5.mail.yahoo.co.jp[XXX.XXX.XXX.XXX]:25, conn_use=4, delay=3.4, delays=0.05/0/0.06/3.3, dsn=5.0.0, status=bounced (host mx5.mail.yahoo.co.jp[XXX.XXX.XXX.XXX] said: 554 delivery error: dd This user doesn't have a yahoo.co.jp account (XXXXX@yahoo.co.jp) [-5] - mta7143.mail.djm.ynwp.yahoo.co.jp (in reply to end of DATA command))",
    "fields": {
      "index_name": "MY_SERVER_NAME-mail"
    },
    "@version": "1",
    "ecs": {
      "version": "1.8.0"
    },
    "@timestamp": "2024-02-05T03:00:35.778Z",
    "event": {
      "original": "Feb  5 12:00:34 MY_SERVER_NAME postfix/smtp[3130871]: 5C3E5C0A4CDC: to=<XXXXX@yahoo.co.jp>, relay=mx5.mail.yahoo.co.jp[XXX.XXX.XXX.XXX]:25, conn_use=4, delay=3.4, delays=0.05/0/0.06/3.3, dsn=5.0.0, status=bounced (host mx5.mail.yahoo.co.jp[XXX.XXX.XXX.XXX] said: 554 delivery error: dd This user doesn't have a yahoo.co.jp account (XXXXX@yahoo.co.jp) [-5] - mta7143.mail.djm.ynwp.yahoo.co.jp (in reply to end of DATA command))"
    }
  },
  "fields": {
    "agent.version.keyword": [
      "7.12.1"
    ],
    "host.architecture.keyword": [
      "x86_64"
    ],
    "host.name.keyword": [
      "MY_SERVER_NAME"
    ],
    "host.hostname": [
      "MY_SERVER_NAME"
    ],
    "host.mac": [
      "00:50:56:89:5a:dc",
      "00:50:56:89:26:34"
    ],
    "agent.hostname.keyword": [
      "MY_SERVER_NAME"
    ],
    "host.ip.keyword": [
      "XXX.XXX.XXX.XXX",
      "XXX.XXX.XXX.XXX"
    ],
    "ecs.version.keyword": [
      "1.8.0"
    ],
    "fields.index_name": [
      "MY_SERVER_NAME-mail"
    ],
    "host.os.version": [
      "8.6 (Green Obsidian)"
    ],
    "host.os.name": [
      "Rocky Linux"
    ],
    "host.id.keyword": [
      "b4a712b2cebb41268aa683d962977dde"
    ],
    "agent.name": [
      "MY_SERVER_NAME"
    ],
    "host.name": [
      "MY_SERVER_NAME"
    ],
    "host.os.version.keyword": [
      "8.6 (Green Obsidian)"
    ],
    "event.original": [
      "Feb  5 12:00:34 MY_SERVER_NAME postfix/smtp[3130871]: 5C3E5C0A4CDC: to=<XXXXX@yahoo.co.jp>, relay=mx5.mail.yahoo.co.jp[XXX.XXX.XXX.XXX]:25, conn_use=4, delay=3.4, delays=0.05/0/0.06/3.3, dsn=5.0.0, status=bounced (host mx5.mail.yahoo.co.jp[XXX.XXX.XXX.XXX] said: 554 delivery error: dd This user doesn't have a yahoo.co.jp account (XXXXX@yahoo.co.jp) [-5] - mta7143.mail.djm.ynwp.yahoo.co.jp (in reply to end of DATA command))"
    ],
    "host.os.type": [
      "linux"
    ],
    "agent.id.keyword": [
      "b6e3e704-1ef9-4bfe-980d-973b6141e02b"
    ],
    "@version.keyword": [
      "1"
    ],
    "input.type": [
      "log"
    ],
    "log.offset": [
      1506901
    ],
    "agent.hostname": [
      "MY_SERVER_NAME"
    ],
    "tags": [
      "kin",
      "mail",
      "beats_input_codec_plain_applied",
      "_grokparsefailure"
    ],
    "host.architecture": [
      "x86_64"
    ],
    "agent.id": [
      "b6e3e704-1ef9-4bfe-980d-973b6141e02b"
    ],
    "ecs.version": [
      "1.8.0"
    ],
    "host.containerized": [
      false
    ],
    "host.hostname.keyword": [
      "MY_SERVER_NAME"
    ],
    "fields.index_name.keyword": [
      "MY_SERVER_NAME-mail"
    ],
    "agent.version": [
      "7.12.1"
    ],
    "host.os.family": [
      ""
    ],
    "input.type.keyword": [
      "log"
    ],
    "tags.keyword": [
      "kin",
      "mail",
      "beats_input_codec_plain_applied",
      "_grokparsefailure"
    ],
    "host.ip": [
      "XXX.XXX.XXX.XXX",
      "XXX.XXX.XXX.XXX"
    ],
    "agent.type": [
      "filebeat"
    ],
    "host.os.kernel.keyword": [
      "4.18.0-372.9.1.el8.x86_64"
    ],
    "host.os.kernel": [
      "4.18.0-372.9.1.el8.x86_64"
    ],
    "@version": [
      "1"
    ],
    "host.os.name.keyword": [
      "Rocky Linux"
    ],
    "host.id": [
      "b4a712b2cebb41268aa683d962977dde"
    ],
    "log.file.path.keyword": [
      "/var/log/maillog"
    ],
    "agent.type.keyword": [
      "filebeat"
    ],
    "agent.ephemeral_id.keyword": [
      "56093b18-ea05-4d11-b240-151139f9af9c"
    ],
    "host.mac.keyword": [
      "00:50:56:89:5a:dc",
      "00:50:56:89:26:34"
    ],
    "agent.name.keyword": [
      "MY_SERVER_NAME"
    ],
    "message": [
      "Feb  5 12:00:34 MY_SERVER_NAME postfix/smtp[3130871]: 5C3E5C0A4CDC: to=<XXXXX@yahoo.co.jp>, relay=mx5.mail.yahoo.co.jp[XXX.XXX.XXX.XXX]:25, conn_use=4, delay=3.4, delays=0.05/0/0.06/3.3, dsn=5.0.0, status=bounced (host mx5.mail.yahoo.co.jp[XXX.XXX.XXX.XXX] said: 554 delivery error: dd This user doesn't have a yahoo.co.jp account (XXXXX@yahoo.co.jp) [-5] - mta7143.mail.djm.ynwp.yahoo.co.jp (in reply to end of DATA command))"
    ],
    "host.os.family.keyword": [
      ""
    ],
    "@timestamp": [
      "2024-02-05T03:00:35.778Z"
    ],
    "host.os.type.keyword": [
      "linux"
    ],
    "host.os.platform.keyword": [
      "rocky"
    ],
    "host.os.platform": [
      "rocky"
    ],
    "log.file.path": [
      "/var/log/maillog"
    ],
    "agent.ephemeral_id": [
      "56093b18-ea05-4d11-b240-151139f9af9c"
    ]
  },
  "ignored_field_values": {
    "message.keyword": [
      "Feb  5 12:00:34 MY_SERVER_NAME postfix/smtp[3130871]: 5C3E5C0A4CDC: to=<XXXXX@yahoo.co.jp>, relay=mx5.mail.yahoo.co.jp[XXX.XXX.XXX.XXX]:25, conn_use=4, delay=3.4, delays=0.05/0/0.06/3.3, dsn=5.0.0, status=bounced (host mx5.mail.yahoo.co.jp[XXX.XXX.XXX.XXX] said: 554 delivery error: dd This user doesn't have a yahoo.co.jp account (XXXXX@yahoo.co.jp) [-5] - mta7143.mail.djm.ynwp.yahoo.co.jp (in reply to end of DATA command))"
    ],
    "event.original.keyword": [
      "Feb  5 12:00:34 MY_SERVER_NAME postfix/smtp[3130871]: 5C3E5C0A4CDC: to=<XXXXX@yahoo.co.jp>, relay=mx5.mail.yahoo.co.jp[XXX.XXX.XXX.XXX]:25, conn_use=4, delay=3.4, delays=0.05/0/0.06/3.3, dsn=5.0.0, status=bounced (host mx5.mail.yahoo.co.jp[XXX.XXX.XXX.XXX] said: 554 delivery error: dd This user doesn't have a yahoo.co.jp account (XXXXX@yahoo.co.jp) [-5] - mta7143.mail.djm.ynwp.yahoo.co.jp (in reply to end of DATA command))"
    ]
  }
}

Badger · February 5, 2024, 4:49am

Your PF pattern is very, very complicated. That makes it really hard to debug. I recommend adding features to a regexp only when you have tested that the previous version has successfully been tested.

I reduced it to

 PF %{POSTFIX}

and it failed right there with a _grokparsefailure tag. That is because MY_SERVER_NAME is not a valid hostname. So you can edit the POSTFIX pattern to use %{NOTSPACE:logsource}. Then instead of adding an alternation of seven different message types, try something like PF %{POSTFIX} (%{POSTFIXSMTP}|%{GREEDYDATA}) and make sure you can parse all the different versions of POSTFIXSMTP. Then go on to test POSTFIXANVIL etc.

its-ogawa · February 5, 2024, 6:21am

You are right, this grok pattern is very complex.
I also understand that it should be tested in advance.

So I am using the Grok Debugger that comes with Kibana.
I have entered the above grok pattern in the Custom Patterns of the Grok Debugger and run it, and it confirms that it succeeds.
The successful results are the results shown in the first post.

Is it possible to succeed in Grok Debugger but fail in Logstash?

Badger · February 5, 2024, 1:28pm

Yes, it is.

Badger · February 5, 2024, 5:20pm

If I change MY_SERVER_NAME to MY-SERVER-NAME, and replace XXX.XXX.XXX.XXX with 1.2.3.4 then your grok patterns work in my logstash

 "relayhost" => "gmail-smtp-in.l.google.com",
     "delay" => "1.2",
       "dsn" => "2.0.0",
       "qid" => "1BF9DC0CC906",
 "logsource" => "MY-SERVER-NAME",
        "to" => "XXXXXX@gmail.com",
    "status" => "sent",
   "relayip" => "1.2.3.4",
 "relayport" => "25",
 "timestamp" => "Feb  5 11:33:10",
       "pid" => "2794930",
   "process" => "smtp",
    "reason" => "(250 2.0.0 OK  1707100390 l10-20020a170902e2ca00b001d741f0eefesi5356753plc.419 - gsmtp)"

its-ogawa · February 6, 2024, 1:28am

I made one mistake. I corrected it as follows.

grok {
  patterns_dir => ["./patterns"]
  match => { "message" => "%{PF}" } # {%PF} -> %{PF}
}

With this response, there is no longer any _grokparsefailure output in tags.
However, the parsed results continue to not appear in fields.

its-ogawa · February 6, 2024, 1:28am

Thank you for trying.

Also, as you pointed out, the server name and IP address can be replaced with the server name and IP address, as I only withheld them in my post just to be safe.

I can see the same results as you did in the Grok Debugger, but not in Logstash.
Could you please tell me your Logstash configuration file (logstash.conf)?

Badger · February 6, 2024, 2:07am

input {
    file { path => "/home/user/foo.txt" sincedb_path => "/dev/null" start_position => beginning }
}

output { stdout { codec => rubydebug { metadata => false } } }
filter {
    mutate { remove_field => [ "event", "host", "log" ] }

    grok {
        patterns_dir => ["/tmp/patterns"]
        match => { "message" => "%{PF}" }
    }
}

ls -l /tmp/patterns/
total 4
-rw-rw-r-- 1 user user 4018 Feb  5 12:14 foo

its-ogawa · February 6, 2024, 2:47am

The description in grok is the same for me.

Can /tmp/patterns/foo be considered the same as the following postfix-grok-patterns?

github.com

padusumilli/postfix-grok/blob/master/postfix-grok-patterns

# Syslog stuff
PROCESS ([\w._\/%-]+)
COMPID postfix\/%{PROCESS:process}(?:\[%{NUMBER:pid}\])?
POSTFIX (?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{COMPID}:

# Postfix stuff
HELO (?:\[%{IP:helo}\]|%{HOSTNAME:helo}|%{DATA:helo})
QUEUEID (?:[A-F0-9]+|NOQUEUE)
EMAILADDRESSPART [a-zA-Z0-9_.+-=:~]+
EMAILADDRESS %{EMAILADDRESSPART:local}@%{EMAILADDRESSPART:remote}
RELAY (?:%{HOSTNAME:relayhost}(?:\[%{IP:relayip}\](?::%{BASE10NUM:relayport}?)?)?)
POSREAL [0-9]+(.[0-9]+)?
DSN %{NONNEGINT}.%{NONNEGINT}.%{NONNEGINT}
STATUS sent|deferred|bounced|expired
PERMERROR 5[0-9]{2}
MESSAGELEVEL reject|warning|error|fatal|panic

POSTFIXSMTPMESSAGE %{MESSAGELEVEL}: %{GREEDYDATA:reason}
POSTFIXACTION discard|dunno|filter|hold|ignore|info|prepend|redirect|replace|reject|warn

This file has been truncated. show original

I know this is not very relevant, but does it make a difference whether the output is to rubydebug or elasticsearch?

Badger · February 6, 2024, 3:09am

Yes, I copied the raw file from github.

It could. If you have dynamic mapping turned off in elasticsearch and your index template does not define them then elasticsearch will not create them.

its-ogawa · February 6, 2024, 4:17am

How do I turn on dynamic mapping?

Adding index.mapper.dynamic: true to elasticsearch.yml does not enable it. (elasticsearch will stop).

Do I need to add a mapping?
I added the mapping in the following way, but I cannot see it on field.

# curl -X PUT "http://localhost:9200/MY-LOG-2024.02.05/_mapping" -H "Content-Type: application/json" -d '{
>   "properties": {
>     "from": {
>       "type": "text",
>       "fields": {
>         "keyword": {
>           "type": "keyword"
>         }
>       }
>     }
>   }
> }'
{"acknowledged":true}

# curl -X GET "http://localhost:9200/MY-LOG-2024.02.05/_mapping?pretty"
{
  "MY-LOG-2024.02.05" : {
    "mappings" : {
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "@version" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "agent" : {
          "properties" : {
            "ephemeral_id" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "hostname" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "id" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "name" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "type" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "version" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "ecs" : {
          "properties" : {
            "version" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "event" : {
          "properties" : {
            "original" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "fields" : {
          "properties" : {
            "index_name" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "from" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword"
            }
          }
        },
        "host" : {
          "properties" : {
            "architecture" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "containerized" : {
              "type" : "boolean"
            },
            "hostname" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "id" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "ip" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "mac" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "name" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "os" : {
              "properties" : {
                "family" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "kernel" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "name" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "platform" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "type" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "version" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                }
              }
            }
          }
        },
        "input" : {
          "properties" : {
            "type" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "log" : {
          "properties" : {
            "file" : {
              "properties" : {
                "path" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                }
              }
            },
            "offset" : {
              "type" : "long"
            }
          }
        },
        "mail" : {
          "properties" : {
            "hostname" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "log-message" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "timestamp" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "message" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "tags" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      }
    }
  }
}

Badger · February 6, 2024, 4:50am

It is on by default. I no longer run elasticsearch, and this is really an ES question. Perhaps @stephenb or @strawgate or someone else who uses the whole stack can help.

stephenb · February 6, 2024, 6:00am

Hi @its-ogawa

Dynamic Mapping is enabled by default; see here

The dynamic parameter controls whether new fields are added dynamically, and accepts the following parameters:

true New fields are added to the mapping (default).

This means when Elastic encounters a new field, it will guess and create a mapping unless you have specifically told the index not to do that.

The setting is Per Index, or you can set it in a template, but it is not set at the cluster level... (AFAIK)

If you want to give Elasticsearch guidance on how to handle Dynamic mappings there are 2 general ways which are described here

Although, after reading through the thread, I am not exactly sure what the issue is...

Sometime fields are not created if first a field comes through as a simple field like

"source" : "127.0.0.1"

and then you trying to index source and as object

"source: {
  "ip" : "127.0.01"
}

the 2nd doc will throw a mapping conflict and will not be indexed...

but if the field is just a new new field it should automatically be added to the index and mapping.

its-ogawa · February 6, 2024, 6:45am

Hi. @stephenb
I see that dynamic mapping is enabled by default.
I appreciate that very much, but I cannot see the fields I have defined in grok filter on Kibana's Discover.
I can get the intended field in json format when parsed in Grok Debugger, but it seems that the field is not created in Elasticsearch.

I thought the problem might be that a new field is added to an already existing index, so I created a new index with an alias, but the result was the same and the intended field was not created.

stephenb · February 6, 2024, 7:07am

My suggestion would be if you're using logstash, do the stdout codec so you can see exactly what's being written to elasticsearch right before it's being written.

It's hard to help if We can't see exactly what you're trying to write to elasticsearch.

Can you show an example of stdout codec output and the same document that's in elasticsearch? And the mismatch?

its-ogawa · February 6, 2024, 7:34am

Thanks for the reply.
I will display the results first.

stdout:

Feb 06 16:20:31 ITS-ELST-01 logstash[689501]: {
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:      "timestamp" => "Feb  6 16:20:24",
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:          "agent" => {
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:              "version" => "7.12.1",
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:                 "name" => "MY-SERVER",
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:             "hostname" => "MY-SERVER",
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:         "ephemeral_id" => "d4d40b29-a2d6-477d-b275-ce3c6bb2be1a",
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:                   "id" => "87215304-c7a1-4914-b057-ef6a368959eb",
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:                 "type" => "filebeat"
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:     },
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:     "@timestamp" => 2024-02-06T07:20:30.758Z,
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:            "ecs" => {
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:         "version" => "1.8.0"
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:     },
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:       "@version" => "1",
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:          "event" => {
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:         "original" => "Feb  6 16:20:24 MY-SERVER postfix/smtp[2989787]: B90FEC014ADE: to=<XXXXXX@XXXX.XXX.XXX>, relay=mx02.au.com[27.86.106.196]:25, delay=0.22, delays=0.06/0.01/0.07/0.07, dsn=2.0.0, status=sent (250 Ok: queued as DB6471600A3)"
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:     },
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:        "message" => "Feb  6 16:20:24 MY-SERVER postfix/smtp[2989787]: B90FEC014ADE: to=<XXXXXX@XXXX.XXX.XXX>, relay=mx02.au.com[27.86.106.196]:25, delay=0.22, delays=0.06/0.01/0.07/0.07, dsn=2.0.0, status=sent (250 Ok: queued as DB6471600A3)",
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:           "host" => {
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:                   "mac" => [
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:             [0] "00:50:56:89:24:4a",
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:             [1] "00:50:56:89:d8:32"
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:         ],
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:                    "os" => {
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:              "version" => "8.6 (Green Obsidian)",
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:                 "name" => "Rocky Linux",
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:               "family" => "",
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:               "kernel" => "4.18.0-372.9.1.el8.x86_64",
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:             "platform" => "rocky",
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:                 "type" => "linux"
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:         },
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:                    "ip" => [
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:             [0] "202.32.104.72",
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:             [1] "192.168.49.12"
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:         ],
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:         "containerized" => false,
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:                  "name" => "MY-SERVER",
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:          "architecture" => "x86_64",
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:              "hostname" => "MY-SERVER",
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:                    "id" => "b4a712b2cebb41268aa683d962977dde"
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:     },
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:           "tags" => [
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:         [0] "kin",
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:         [1] "mail",
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:         [2] "beats_input_codec_plain_applied"
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:     ],
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:         "fields" => {
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:         "index_name" => "MY-MAIL"
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:     },
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:            "log" => {
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:           "file" => {
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:             "path" => "/var/log/maillog"
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:         },
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:         "offset" => 3624292
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:     },
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:      "logsource" => "MY-SERVER",
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:            "pid" => "2989787",
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:          "input" => {
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:         "type" => "log"
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:     },
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]:        "process" => "smtp"
Feb 06 16:20:31 ITS-ELST-01 logstash[689501]: }

Elasticsearch(By Kibana)

{
  "_index": "MY-MAIL-2024.02.06",
  "_id": "fcJKfY0BX2moE9zJNd6f",
  "_version": 1,
  "_score": 0,
  "_source": {
    "timestamp": "Feb  6 16:20:24",
    "agent": {
      "version": "7.12.1",
      "name": "MY-SERVER",
      "hostname": "MY-SERVER",
      "ephemeral_id": "d4d40b29-a2d6-477d-b275-ce3c6bb2be1a",
      "id": "87215304-c7a1-4914-b057-ef6a368959eb",
      "type": "filebeat"
    },
    "@timestamp": "2024-02-06T07:20:30.758Z",
    "ecs": {
      "version": "1.8.0"
    },
    "@version": "1",
    "event": {
      "original": "Feb  6 16:20:24 MY-SERVER postfix/smtp[2989787]: B90FEC014ADE: to=<XXXXXX@XXX.XXX.XXX>, relay=mx02.au.com[27.86.106.196]:25, delay=0.22, delays=0.06/0.01/0.07/0.07, dsn=2.0.0, status=sent (250 Ok: queued as DB6471600A3)"
    },
    "message": "Feb  6 16:20:24 MY-SERVER postfix/smtp[2989787]: B90FEC014ADE: to=<XXXXXX@XXX.XXX.XXX>, relay=mx02.au.com[27.86.106.196]:25, delay=0.22, delays=0.06/0.01/0.07/0.07, dsn=2.0.0, status=sent (250 Ok: queued as DB6471600A3)",
    "host": {
      "mac": [
        "00:50:56:89:24:4a",
        "00:50:56:89:d8:32"
      ],
      "os": {
        "version": "8.6 (Green Obsidian)",
        "name": "Rocky Linux",
        "family": "",
        "kernel": "4.18.0-372.9.1.el8.x86_64",
        "platform": "rocky",
        "type": "linux"
      },
      "ip": [
        "202.32.104.72",
        "XXX.XXX.XXX.XXX"
      ],
      "containerized": false,
      "name": "MY-SERVER",
      "architecture": "x86_64",
      "hostname": "MY-SERVER",
      "id": "b4a712b2cebb41268aa683d962977dde"
    },
    "tags": [
      "kin",
      "mail",
      "beats_input_codec_plain_applied"
    ],
    "fields": {
      "index_name": "MY-MAIL"
    },
    "log": {
      "file": {
        "path": "/var/log/maillog"
      },
      "offset": 3624292
    },
    "logsource": "MY-SERVER",
    "pid": "2989787",
    "input": {
      "type": "log"
    },
    "process": "smtp"
  },
  "fields": {
    "agent.version.keyword": [
      "7.12.1"
    ],
    "host.architecture.keyword": [
      "x86_64"
    ],
    "host.name.keyword": [
      "MY-SERVER"
    ],
    "pid": [
      "2989787"
    ],
    "host.hostname": [
      "MY-SERVER"
    ],
    "host.mac": [
      "00:50:56:89:24:4a",
      "00:50:56:89:d8:32"
    ],
    "agent.hostname.keyword": [
      "MY-SERVER"
    ],
    "ecs.version.keyword": [
      "1.8.0"
    ],
    "host.ip.keyword": [
      "202.32.104.72",
      "XXX.XXX.XXX.XXX"
    ],
    "fields.index_name": [
      "MY-MAIL"
    ],
    "host.os.version": [
      "8.6 (Green Obsidian)"
    ],
    "host.os.name": [
      "Rocky Linux"
    ],
    "agent.name": [
      "MY-SERVER"
    ],
    "host.id.keyword": [
      "b4a712b2cebb41268aa683d962977dde"
    ],
    "host.name": [
      "MY-SERVER"
    ],
    "host.os.version.keyword": [
      "8.6 (Green Obsidian)"
    ],
    "logsource.keyword": [
      "MY-SERVER"
    ],
    "event.original": [
      "Feb  6 16:20:24 MY-SERVER postfix/smtp[2989787]: B90FEC014ADE: to=<XXXXXX@XXX.XXX.XXX>, relay=mx02.au.com[27.86.106.196]:25, delay=0.22, delays=0.06/0.01/0.07/0.07, dsn=2.0.0, status=sent (250 Ok: queued as DB6471600A3)"
    ],
    "host.os.type": [
      "linux"
    ],
    "agent.id.keyword": [
      "87215304-c7a1-4914-b057-ef6a368959eb"
    ],
    "@version.keyword": [
      "1"
    ],
    "input.type": [
      "log"
    ],
    "log.offset": [
      3624292
    ],
    "agent.hostname": [
      "MY-SERVER"
    ],
    "tags": [
      "kin",
      "mail",
      "beats_input_codec_plain_applied"
    ],
    "host.architecture": [
      "x86_64"
    ],
    "agent.id": [
      "87215304-c7a1-4914-b057-ef6a368959eb"
    ],
    "ecs.version": [
      "1.8.0"
    ],
    "host.containerized": [
      false
    ],
    "message.keyword": [
      "Feb  6 16:20:24 MY-SERVER postfix/smtp[2989787]: B90FEC014ADE: to=<XXXXXX@XXX.XXX.XXX>, relay=mx02.au.com[27.86.106.196]:25, delay=0.22, delays=0.06/0.01/0.07/0.07, dsn=2.0.0, status=sent (250 Ok: queued as DB6471600A3)"
    ],
    "fields.index_name.keyword": [
      "MY-MAIL"
    ],
    "host.hostname.keyword": [
      "MY-SERVER"
    ],
    "agent.version": [
      "7.12.1"
    ],
    "host.os.family": [
      ""
    ],
    "input.type.keyword": [
      "log"
    ],
    "tags.keyword": [
      "kin",
      "mail",
      "beats_input_codec_plain_applied"
    ],
    "process.keyword": [
      "smtp"
    ],
    "timestamp.keyword": [
      "Feb  6 16:20:24"
    ],
    "host.ip": [
      "202.32.104.72",
      "XXX.XXX.XXX.XXX"
    ],
    "pid.keyword": [
      "2989787"
    ],
    "agent.type": [
      "filebeat"
    ],
    "host.os.kernel.keyword": [
      "4.18.0-372.9.1.el8.x86_64"
    ],
    "host.os.kernel": [
      "4.18.0-372.9.1.el8.x86_64"
    ],
    "@version": [
      "1"
    ],
    "host.os.name.keyword": [
      "Rocky Linux"
    ],
    "host.id": [
      "b4a712b2cebb41268aa683d962977dde"
    ],
    "log.file.path.keyword": [
      "/var/log/maillog"
    ],
    "agent.type.keyword": [
      "filebeat"
    ],
    "timestamp": [
      "Feb  6 16:20:24"
    ],
    "agent.ephemeral_id.keyword": [
      "d4d40b29-a2d6-477d-b275-ce3c6bb2be1a"
    ],
    "process": [
      "smtp"
    ],
    "host.mac.keyword": [
      "00:50:56:89:24:4a",
      "00:50:56:89:d8:32"
    ],
    "agent.name.keyword": [
      "MY-SERVER"
    ],
    "logsource": [
      "MY-SERVER"
    ],
    "message": [
      "Feb  6 16:20:24 MY-SERVER postfix/smtp[2989787]: B90FEC014ADE: to=<XXXXXX@XXX.XXX.XXX>, relay=mx02.au.com[27.86.106.196]:25, delay=0.22, delays=0.06/0.01/0.07/0.07, dsn=2.0.0, status=sent (250 Ok: queued as DB6471600A3)"
    ],
    "host.os.family.keyword": [
      ""
    ],
    "@timestamp": [
      "2024-02-06T07:20:30.758Z"
    ],
    "host.os.type.keyword": [
      "linux"
    ],
    "host.os.platform": [
      "rocky"
    ],
    "host.os.platform.keyword": [
      "rocky"
    ],
    "event.original.keyword": [
      "Feb  6 16:20:24 MY-SERVER postfix/smtp[2989787]: B90FEC014ADE: to=<XXXXXX@XXX.XXX.XXX>, relay=mx02.au.com[27.86.106.196]:25, delay=0.22, delays=0.06/0.01/0.07/0.07, dsn=2.0.0, status=sent (250 Ok: queued as DB6471600A3)"
    ],
    "log.file.path": [
      "/var/log/maillog"
    ],
    "agent.ephemeral_id": [
      "d4d40b29-a2d6-477d-b275-ce3c6bb2be1a"
    ]
  }
}

its-ogawa · February 6, 2024, 7:37am

Grok Debugger:

{
  "reason": "(250 Ok: queued as DB6471600A3)",
  "process": "smtp",
  "relayport": "25",
  "relayip": "27.86.106.196",
  "relayhost": "mx02.au.com",
  "pid": "2989787",
  "logsource": "MY-SERVER",
  "qid": "B90FEC014ADE",
  "delay": "0.22",
  "to": "XXXXXX@XXX.XXX.XXX",
  "dsn": "2.0.0",
  "timestamp": "Feb  6 16:20:24",
  "status": "sent"
}

What we found here is that Logstash is not doing the parsing we expect.
I want a field like the Grok Debugger results.
For example, to, status, etc.

stephenb · February 6, 2024, 2:37pm

Hi @its-ogawa

First, I don't think this has anything to do with dynamic mapping as it does not look like you're sending that data to Elastic in the first place.

Second, I don't see on your logstash output doesn't even look like the grok statements being called because should get a grok parse error if it is and failing. Maybe your if statement is not being calculated correctly.

I would isolate and simplify your problem for debugging

What I would do is take a couple of lines that you know need to be parsed... Put three lines in a file with the log lines you want to grok.

Make a very simple logstash pipeline that only does the grok, take out the conditionals.

Check the output.

Start with a simple grok.. add complexity...

Test repeat.

This looks like what you were doing at the top... perhaps I think you were closer there...

It's not clear what grok debugger you are talking about.