Cannot parse empty date

[stefanocog]

Hi, I have a log that has a json field inside that can have empty fields, specifically I have a date field, the log can be like this

"2022-11-28 09:24:46:705"|"+0100"|"transId: xxxxxx"|"resId: xxxxxx"|"1.1.1.1"|"https://example.com/xxxxxx"|"HTTP/1.1"|"EXE"|"GET"|"404"|"{"codCpi":"","codFis": "","codSap": "","codState": "","datBirth": "", "xml": ""}"|"token : xxxx"

Have this rule for match:

filter {
  grok {
    match => { "message" => "\"%{TIMESTAMP_ISO8601:timestamp}\"\|\"%{DATA:tz}\"\|\"transId: %{GREEDYDATA:transactionId}\"\|\"resId: %{GREEDYDATA:responseId}\"\|\"%{IP:ip}\"\|\"%{GREEDYDATA:url}\"\|\"%{DATA:httpver}\"\|\"%{DATA:exe}\"\|\"%{WORD:httpverb}\"\|\"%{GREEDYDATA:httpCodeResponse}\"\|\"%{DATA:contentRequest}\"\|\"%{GREEDYDATA:token}\"" }
  }
  json {
    source => "contentRequest"
  }
}

With this match i have this error:

"error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [datBirth] of type [date] in document with id 'VSFI54QBv1ziCbBLadWr'. Preview of field's value: ''", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"cannot parse empty date"}}}}

Instead, there is no error if the field datBirth is filled in

[carly.richmond]

To confirm, are you only receiving this error for events where datBirth is empty, and parsing is successful when it's populated?

If so you could try specifying zero or one occurrences using the ? operator, as per the below:

Making a part in the grok expression optional

Hi all, Each line in my log file does not contain a source IP adderss so the "message" => "%{TIMESTAMP_ISO8601:timestamp}\s+%{LOGLEVEL:loglevel}\s+%{THREAD:thread}\s+(?:%{IP:ip})" so it returns a "no mach" - from grok debugger. As result the the fields that I'm expecting to be created by the grok{} is not happening. So we tried to make the pattern for IP in this expression something like : match => {"message" => "%{TIMESTAMP_ISO8601:timestamp}\s+%{LOGLEVEL:loglevel}\s+%{THREAD:thread}…

[stefanocog]

I confirm that if the field is filled in, it works.
Unfortunately it is a service of which I have no control, I don't think it can be changed

[leandrojmp (Leandro Pereira)]

You can remove the field if it is empty.

if [dateBirth] == "" {
    mutate {
        remove_field => ["dateBirth"]
    }
}

[stefanocog]

Sorry I did not understand
Seems to work with this rule

filter {
  grok {
    match => { "message" => "\"%{TIMESTAMP_ISO8601:timestamp}\"\|\"%{DATA:tz}\"\|\"transId: %{GREEDYDATA:transactionId}\"\|\"resId: %{GREEDYDATA:responseId}\"\|\"%{IP:ip}\"\|\"%{GREEDYDATA:url}sap/API/SAP/codFiscale\/%{GREEDYDATA:codicefiscale}\"\|\"%{DATA:httpver}\"\|\"%{DATA:gruppo}\"\|\"%{WORD:httpverb}\"\|\"%{GREEDYDATA:httpCodeResponse}\"\|\"(?<contentRequest>{.*})\"\|\"%{GREEDYDATA:token}\"" }
  }
  json {
    source => "contentRequest"
  }
}

Thanks!