Hi all,
I hope someone can help me. I am really new to Elasticsearch so any guidance would be really appreciated.
I have been battling with AWS Elasticsearch and Kinesis FIrehose Agent (this reads in a application log file) and i think I think I am almost there but I have hit a blocker.
> {"type":"mapper_parsing_exception","reason":"failed to parse field [timestamp] of type [date] in document with id \u002749606120302139348740181849332896261353140434064317087746.0\u0027. Preview of field\u0027s value: \u00272020-04-16 17:48:25,839\u0027","caused_by":{"type":"illegal_argument_exception","reason":"failed to parse date field [2020-04-16 17:48:25,839] with format [strict_date_optional_time||epoch_millis]","caused_by":{"type":"date_time_parse_exception","reason":"Failed to parse with all enclosed parsers"}}}
The data in question is
2020-04-28 19:48:25,244|OAuth| 1234567890| 127.0.0.1 | | pa_customer| OAuth20| localhost| AS| success| ProductHoldingAFM2FA| | 2596
The error message hints at \u0027 which i think is an apostrophe but the log has no aprostrophe in it?
All the data from the kinesis comes with data types defaulting as text i changed it in elasticsearch with the following template.
` PUT /_template/testfed-t01
{
"index_patterns": [
"test*"
],
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1
},
"mappings": {
"_source": {"enabled": true},
"properties": {
"action": {"type": "text"},
"authenticationtype": {"type": "text"},
"device": {"type": "keyword"},
"duration": {"type":"integer"},
"hosttype": {"type": "text"},
"ipaddress": {"type": "keyword"},
"message": {"type":"text"},
"providertype": {"type": "text"},
"result": {"type": "text"},
"timestamp": {
"type": "date",
"format":"yyyy-MM-dd HH:mm:ss,SSS"
},
"typemfa": {"type":"text"},
"unknown1": {"type": "text"},
"unknown2": {"type": "text"}
}
}
}`
My data is structured as:
> 2020-04-28 19:48:25,839|OAuth| 1234567890| 127.0.0.1 | | pa_customer| OAuth20| localhost| AS| success| | | 521 > 2020-04-28 19:38:25,839|OAuth| 1234567890| 127.0.0.1 | | pa_customer| OAuth20| localhost| AS| success| | | 521
My kinesisagent.json looks like :
{ "checkpointFile": "/opt/aws-kinesis-agent/run/checkpoints", "cloudwatch.endpoint": "https://monitoring.eu-west-2.amazonaws.com", "firehose.endpoint": "firehose.eu-west-2.amazonaws.com", "awsAccessKeyId": "AKIFREDGJAMDNDREMSJ", "awsSecretAccessKey": "2xxsx4TDEs34WQ0UaMpFHwu4h+FAKEF8VxedtPMADZ", "flows": [ { "filePattern": "/data/applicatee-10.0.0/applicatee/log/testaudit01.log", "initialPosition": "START_OF_FILE", "deliveryStream": "TEST-APPlicatee-AuditLog-Stream", "dataProcessingOptions": [ { "optionName": "CSVTOJSON", "customFieldNames": [ "timestamp", "action", "unknown1", "ipaddress", "unknown2","device", "authenticationtype", "hosttype","providertype", "result", "typemfa", "message", "duration" ], "delimiter": "\\|" } ] } ] }
Any help would be greatly appreciated.
Thanks again in advance for your time and help.
yoyomonkey