Hello,
I have log messages with a mytimesmap
field. This field contains microseconds precision RFC3339/ISO8601 (UTC) style timestamp like 2021-03-14T13:25:49.008906Z
.
So I'd like to overwrite @timestamp
field with mytimestamp
fields content with the timestamp processor
.
Here is the relevant Filebeat config:
processors:
- decode_json_fields:
fields: ["message"]
process_array: true
target: ""
add_error_key: true
overwrite_keys: true
- timestamp:
field: mytimestamp
layouts:
- '2020-01-02T12:34:56.999999Z'
test:
- '2020-01-02T12:34:56.123456Z'
if I start the Filebeat with the config above I get an error message:
Exiting: error initializing processors: failed to parse test timestamp: failed parsing time field mytimestamp='2020-01-02T12:34:56.123456Z'
So Filebeat failed to start because timestamp processor can't validate the timestamp but there is no explanation why not.
As per Filebeat documentation (Timestamp | Filebeat Reference [7.11] | Elastic) the test
part of the timestamp processor config is not mandatory so I commented it out.
After that Filebeat seems to have worked fine except it did not overwrite @timestamp
field.
So I've set logging.level
to debug
and these error messages appears in the log for every log message
2021-03-14T14:25:49.979+0100 DEBUG [processor.timestamp] timestamp/timestamp.go:173 Failure parsing time field. {"error": "failed parsing time field mytimestamp='2021-03-14T13:25:49.008906Z'", "errorCauses": [{"error": "failed using layout [2020-01-02T12:34:56.999999Z] cannot parse [-03-14T13:25:49.008906Z] as [0-]"}]}
2021-03-14T14:25:49.979+0100 DEBUG [processors] processing/processors.go:128 Fail to apply processor global{decode_json_fields=message, timestamp=[field=mytimestamp, target_field=@timestamp, timezone=UTC, layouts=[2020-01-02T12:34:56.999999Z]]}: failed parsing time field mytimestamp='2021-03-14T13:25:49.008906Z'
The input log timestamp was 2021-03-14T13:25:49.008906Z
but the timestamp processor somehow lost/chop off the years (or the first 4 ?) character of the timestamp.
Why timestamp processor truncates the timestamp?
How should i construct the layout
to allow a plain and simple 2021-03-14T13:25:49.008906Z
timestamp?
Filebeat 7.11.2, Elasticsearch 7.11.2, Kibana 7.11.2.
Note:
On Elasticsearch side I've set both @timestamp
and mytimestamp
fields type to date_nanos
with a mapping template:
curl -X PUT "http://$1:9200/_index_template/myindex_date_nano_template" -H 'Content-Type: application/json' -d'
{
"priority": 200,
"index_patterns": ["myindex-*"],
"template":
{
"mappings":
{
"properties":
{
"@timestamp":
{
"type": "date_nanos"
},
"mytimestamp":
{
"type": "date_nanos"
}
}
}
}
}
'
after applying that template mytimestamp
shows up correctly (with the microsconds parts) in Kibana but @timestamp
still not overwritten with the content of mytimestamp
.