Help parsing a date field

Elastic Stack Elasticsearch
1

Hello, my source data has the following dates, which i am struggling to create the correct field mapping.

"notBefore": "Jun 9 00:31:02 2020 GMT", "notAfter": "Jun 9 00:31:02 2025 GMT"

i have tried various combinations, but basically looking like this, and i just can't get it right.

"notBefore": {
            "type": "date",
            "format": "MMM  dd hh:mm:ss yyyy ZZZ",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }

The errors are obvious

{"index":{"_index":"sslscan-2020","_type":"_doc","_id":"UHlkLHMBb4m0aeo6mAc9","status":400,"error":{"type":"mapper_parsing_exception","reason":"failed to parse field [certificateChain.notBefore] of type [date] in document with id 'UHlkLHMBb4m0aeo6mAc9'. Preview of field's value: 'Dec  8 00:00:00 2018 GMT'","caused_by":{"type":"illegal_argument_exception","reason":"failed to parse date field [Dec  8 00:00:00 2018 GMT] with format [strict_date_optional_time||epoch_millis]","caused_by":{"type":"date_time_parse_exception","reason":"Failed to parse with all enclosed parsers"}}}}}
2

according to the documentation custom date format is supported

I would try something like

MMM d HH:mm:ss uuuu z

Did not test

3

Hi @AClerk - thank you for the reply. Unfortunately, I tried this and still get parsing errors. I think i will solve this another way and shape the date correctly prior to sending it to ES.

4

I have now instead used https://github.com/PaulSec/crt.sh to pull certificates straight from CTL, and POST them to ES using the python bulk helper, ref (https://kb.objectrocket.com/elasticsearch/how-to-use-python-helpers-to-bulk-load-data-into-an-elasticsearch-index)

Works like a dream.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.