# Timestamp processor truncates timestamp and fails to parse
**URL:** https://discuss.elastic.co/t/timestamp-processor-truncates-timestamp-and-fails-to-parse/267190
**Category:** Beats
**Tags:** filebeat
**Created:** [March 14, 2021, 2:25pm UTC](https://discuss.elastic.co/t/timestamp-processor-truncates-timestamp-and-fails-to-parse/267190 "2021-03-14T14:25:30Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@6NMgfDwZ3](https://discuss.elastic.co/u/6NMgfDwZ3)
#### Post date: [March 14, 2021, 2:25pm UTC](https://discuss.elastic.co/t/timestamp-processor-truncates-timestamp-and-fails-to-parse/267190/1 "2021-03-14T14:25:30Z")
Hello,
I have log messages with a `mytimesmap` field. This field contains microseconds precision RFC3339/ISO8601 (UTC) style timestamp like `2021-03-14T13:25:49.008906Z`.
So I'd like to overwrite `@timestamp` field with `mytimestamp` fields content with the `timestamp processor`.
Here is the relevant Filebeat config:
```auto
processors:
- decode_json_fields:
fields: ["message"]
process_array: true
target: ""
add_error_key: true
overwrite_keys: true
- timestamp:
field: mytimestamp
layouts:
- '2020-01-02T12:34:56.999999Z'
test:
- '2020-01-02T12:34:56.123456Z'
```
if I start the Filebeat with the config above I get an error message:
```auto
Exiting: error initializing processors: failed to parse test timestamp: failed parsing time field mytimestamp='2020-01-02T12:34:56.123456Z'
```
So Filebeat failed to start because timestamp processor can't validate the timestamp but there is no explanation why not.
As per Filebeat documentation ([Timestamp | Filebeat Reference [7.11] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/processor-timestamp.html)) the `test` part of the timestamp processor config is not mandatory so I commented it out.
After that Filebeat seems to have worked fine except it did not overwrite `@timestamp` field.
So I've set `logging.level` to `debug` and these error messages appears in the log for every log message
```auto
2021-03-14T14:25:49.979+0100 DEBUG [processor.timestamp] timestamp/timestamp.go:173 Failure parsing time field. {"error": "failed parsing time field mytimestamp='2021-03-14T13:25:49.008906Z'", "errorCauses": [{"error": "failed using layout [2020-01-02T12:34:56.999999Z] cannot parse [-03-14T13:25:49.008906Z] as [0-]"}]}
2021-03-14T14:25:49.979+0100 DEBUG [processors] processing/processors.go:128 Fail to apply processor global{decode_json_fields=message, timestamp=[field=mytimestamp, target_field=@timestamp, timezone=UTC, layouts=[2020-01-02T12:34:56.999999Z]]}: failed parsing time field mytimestamp='2021-03-14T13:25:49.008906Z'
```
The input log timestamp was `2021-03-14T13:25:49.008906Z` but the timestamp processor somehow lost/chop off the years (or the first 4 ?) character of the timestamp.
Why timestamp processor truncates the timestamp?
How should i construct the `layout` to allow a plain and simple `2021-03-14T13:25:49.008906Z` timestamp?
Filebeat 7.11.2, Elasticsearch 7.11.2, Kibana 7.11.2.
Note:
On Elasticsearch side I've set both `@timestamp` and `mytimestamp` fields type to `date_nanos` with a mapping template:
```auto
curl -X PUT "http://$1:9200/_index_template/myindex_date_nano_template" -H 'Content-Type: application/json' -d'
{
"priority": 200,
"index_patterns": ["myindex-*"],
"template":
{
"mappings":
{
"properties":
{
"@timestamp":
{
"type": "date_nanos"
},
"mytimestamp":
{
"type": "date_nanos"
}
}
}
}
}
'
```
after applying that template `mytimestamp` shows up correctly (with the microsconds parts) in Kibana but `@timestamp` still not overwritten with the content of `mytimestamp`.
---
### Author:  [@shaunak](https://discuss.elastic.co/u/shaunak)
#### Post date: [March 15, 2021, 7:32pm UTC](https://discuss.elastic.co/t/timestamp-processor-truncates-timestamp-and-fails-to-parse/267190/2 "2021-03-15T19:32:11Z")
Please read about `layouts` in the [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/processor-timestamp.html) for the `timestamp` processor. They are based on a **specific** date; any arbitrary date cannot be used when specifying the layout.
For your case, try this:
```auto
layouts:
- "2006-01-02T15:04:05.999999999Z07:00"
```
Shaunak
---
### Author:  [@Shawn\_McKee](https://discuss.elastic.co/u/Shawn_McKee)
#### Post date: [March 16, 2021, 5:41pm UTC](https://discuss.elastic.co/t/timestamp-processor-truncates-timestamp-and-fails-to-parse/267190/3 "2021-03-16T17:41:13Z")
We are seeing the same problem. Elasticsearch 7.11.2, Filebeat 7.11.2 and Kibana 7.11.2
In filebeat we have this timestamp processor configured:
```auto
- timestamp:
field: timedata
ignore_failure: false
timezone: "America/Detroit"
layouts:
- '2020-12-01T15:08:05'
- '2020-12-01T15:08:05.000'
- '2020-12-01T15:08:05.000Z04:00'
test:
- '2006-01-02T15:04:05.000Z07:00'
```
Filebeat log file (in 'debug') shows the same year truncation reported above:
```auto
2021-03-16T13:32:33.603-0400 DEBUG [processor.timestamp] timestamp/timestamp.go:173 Failure parsing time field. {"error": "failed parsing time field timedata='2006-01-02T15:04:05.000Z07:00'", "errorCauses": [{"error": "failed using layout [2020-12-01T15:08:05] cannot parse [-01-02T15:04:05.000Z07:00] as [0-]"}, {"error": "failed using layout [2020-12-01T15:08:05.000] cannot parse [-01-02T15:04:05.000Z07:00] as [0-]"}, {"error": "failed using layout [2020-12-01T15:08:05.000Z04:00] cannot parse [-01-02T15:04:05.000Z07:00] as [0-]"}]}
2021-03-16T13:32:33.603-0400 INFO instance/beat.go:386 filebeat stopped.
2021-03-16T13:32:33.623-0400 ERROR instance/beat.go:971 Exiting: error initializing processors: failed to parse test timestamp: failed parsing time field timedata='2006-01-02T15:04:05.000Z07:00'
```
Note the logging starts with the right value:
**timedata='2006-01-02T15:04:05.000Z07:00'**
But then, as it tries to pattern match is uses a string with the year removed:
**cannot parse [-01-02T15:04:05.000Z07:00]**
Seems like there is a bug here? Can someone check?
Thanks - Shawn
---
### Author:  [@sandman0](https://discuss.elastic.co/u/sandman0)
#### Post date: [March 16, 2021, 8:43pm UTC](https://discuss.elastic.co/t/timestamp-processor-truncates-timestamp-and-fails-to-parse/267190/4 "2021-03-16T20:43:04Z")
I am seeing the same behaviour as @Shawn_McKee on Filebeat 7.11.1.
---
### Author:  [@6NMgfDwZ3](https://discuss.elastic.co/u/6NMgfDwZ3)
#### Post date: [March 16, 2021, 10:12pm UTC](https://discuss.elastic.co/t/timestamp-processor-truncates-timestamp-and-fails-to-parse/267190/5 "2021-03-16T22:12:14Z")
@shaunak Thank you so much, at least I'm one step ahead because Filebeat does not log any error messages, and the "test dates" passes the test.
Relevant `filebeat.yml` part:
```auto
processors:
- decode_json_fields:
fields: ["message"]
process_array: true
target: ""
add_error_key: true
overwrite_keys: true
- timestamp:
field: mytimestamp
layouts:
- "2006-01-02T15:04:05.999999999Z07:00"
test:
- "2020-01-02T12:34:56.123456Z"
- "2021-03-15T22:29:34.213223Z"
```
Apart from that Filebeat still not overwriting `@timestamp` field but I'll open an other discussion about it.
* * *
@Shawn_McKee @sandman0 you have to fiddle with the `Mon Jan 2 15:04:05 MST 2006` timestamp and transform that timestamp to an other form what your needs. This madness comes from the `Go` `time` package: [time ยท pkg.go.dev](https://pkg.go.dev/time?utm_source=godoc#pkg-constants) More explanation: [Go Date and Time Formatting](https://flaviocopes.com/go-date-time-format/)
Sum: You can't write a random datetime to the `layout` field you have to stick to the specific timestamp (above) so the underlaying Go library can match and confirm the timestamp pieces of your `layout` samples
---
### Author:  [@Shawn\_McKee](https://discuss.elastic.co/u/Shawn_McKee)
#### Post date: [March 17, 2021, 12:47am UTC](https://discuss.elastic.co/t/timestamp-processor-truncates-timestamp-and-fails-to-parse/267190/6 "2021-03-17T00:47:07Z")
Thanks, that worked. Here is what I ended up with:
```auto
- timestamp:
field: timedata
ignore_failure: false
timezone: "America/Detroit"
layouts:
- '2006-01-02T15:04:05.999999999'
- '2006-01-02T15:04:05.000'
- '2006-01-02T15:04:05.000-07:00'
- '2006-01-02T15:04:05'
test:
- '2016-01-02T19:04:05'
- '2016-01-02T19:04:05.241'
- '2016-01-02T19:04:05.241-04:00'
```
The @timestamp field is being correctly updated as well. Thanks again,
Shawn
---
### Author:  [@sandman0](https://discuss.elastic.co/u/sandman0)
#### Post date: [March 17, 2021, 3:09am UTC](https://discuss.elastic.co/t/timestamp-processor-truncates-timestamp-and-fails-to-parse/267190/7 "2021-03-17T03:09:00Z")
Yes, works for me now as well. I had no idea about that whole fixed reference date thing in go.
Thank you @6NMgfDwZ3 !
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 14, 2021, 5:09am UTC](https://discuss.elastic.co/t/timestamp-processor-truncates-timestamp-and-fails-to-parse/267190/8 "2021-04-14T05:09:36Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.