AWS Elasticsearch : mapper_parsing_exception for type date from Kinesis Firehose using index_template

Elastic Stack Elasticsearch
1

Hi all,

I hope someone can help me. I am really new to Elasticsearch so any guidance would be really appreciated.

I have been battling with AWS Elasticsearch and Kinesis FIrehose Agent (this reads in a application log file) and i think I think I am almost there but I have hit a blocker.

> {"type":"mapper_parsing_exception","reason":"failed to parse field [timestamp] of type [date] in document with id \u002749606120302139348740181849332896261353140434064317087746.0\u0027. Preview of field\u0027s value: \u00272020-04-16 17:48:25,839\u0027","caused_by":{"type":"illegal_argument_exception","reason":"failed to parse date field [2020-04-16 17:48:25,839] with format [strict_date_optional_time||epoch_millis]","caused_by":{"type":"date_time_parse_exception","reason":"Failed to parse with all enclosed parsers"}}}

The data in question is

2020-04-28 19:48:25,244|OAuth| 1234567890| 127.0.0.1 | | pa_customer| OAuth20| localhost| AS| success| ProductHoldingAFM2FA| | 2596

The error message hints at \u0027 which i think is an apostrophe but the log has no aprostrophe in it?

All the data from the kinesis comes with data types defaulting as text i changed it in elasticsearch with the following template.

` PUT /_template/testfed-t01
{
"index_patterns": [
"test*"
],
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1
},
"mappings": {

      "_source": {"enabled": true}, 
      "properties": {
      "action": {"type": "text"}, 
      "authenticationtype": {"type": "text"}, 
      "device": {"type": "keyword"}, 
      "duration": {"type":"integer"}, 
      "hosttype": {"type": "text"}, 
      "ipaddress": {"type": "keyword"}, 
      "message": {"type":"text"}, 
      "providertype": {"type": "text"},
      "result": {"type": "text"}, 
      "timestamp": {
        "type": "date",
        "format":"yyyy-MM-dd HH:mm:ss,SSS"
      }, 
      "typemfa": {"type":"text"}, 
      "unknown1": {"type": "text"},
      "unknown2": {"type": "text"}

    }
  }
}`

My data is structured as:

> 2020-04-28 19:48:25,839|OAuth| 1234567890| 127.0.0.1 | | pa_customer| OAuth20| localhost| AS| success| | | 521 > 2020-04-28 19:38:25,839|OAuth| 1234567890| 127.0.0.1 | | pa_customer| OAuth20| localhost| AS| success| | | 521

My kinesisagent.json looks like :

{ "checkpointFile": "/opt/aws-kinesis-agent/run/checkpoints", "cloudwatch.endpoint": "https://monitoring.eu-west-2.amazonaws.com", "firehose.endpoint": "firehose.eu-west-2.amazonaws.com", "awsAccessKeyId": "AKIFREDGJAMDNDREMSJ", "awsSecretAccessKey": "2xxsx4TDEs34WQ0UaMpFHwu4h+FAKEF8VxedtPMADZ", "flows": [ { "filePattern": "/data/applicatee-10.0.0/applicatee/log/testaudit01.log", "initialPosition": "START_OF_FILE", "deliveryStream": "TEST-APPlicatee-AuditLog-Stream", "dataProcessingOptions": [ { "optionName": "CSVTOJSON", "customFieldNames": [ "timestamp", "action", "unknown1", "ipaddress", "unknown2","device", "authenticationtype", "hosttype","providertype", "result", "typemfa", "message", "duration" ], "delimiter": "\\|" } ] } ] }

Any help would be greatly appreciated.

Thanks again in advance for your time and help.

yoyomonkey