# Help with date-time conversion from Filebeat **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [December 18, 2023, 3:42pm UTC](https://discuss.elastic.co/t/help-with-date-time-conversion-from-filebeat/349573 "2023-12-18T15:42:19Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![bobus](https://avatars.discourse-cdn.com/v4/letter/b/e9bcb4/32.png) [@bobus](https://discuss.elastic.co/u/bobus)\ **Post date:** [December 18, 2023, 3:42pm UTC](https://discuss.elastic.co/t/help-with-date-time-conversion-from-filebeat/349573/1 "2023-12-18T15:42:19Z")
I'm sending log data from Filebeat (running on Kubernetes) to Graylog/Elasticsearch. I need to ensure that the date-time field inside the JSON message block that is part of the log entry has this format: 2023-12-11T23:23:22.000Z Right now the middle "T" is missing. So I'm using this config: ```auto filebeat.yml: |- filebeat.inputs: - type: container paths: - /var/log/containers/*.log processors: - add_kubernetes_metadata: host: ${NODE_NAME} matchers: - logs_path: logs_path: "/var/log/containers/" - decode_json_fields: fields: ["message"] target: "_message" overwrite_keys: true add_error_key: true - convert: fields: - {from: "_message_timestamp", type: "string", layout: "2006-01-02T15:04:05.999"} ignore_missing: true ``` The "message" JSON block is properly parsed and its fields extracted into new fields with the "\_message" prefix, but the conversion does not take place and we still get the error from Elasticsearch. In fact, when, just for testing, I replaced that kind-of-complex conversion with one that just changes the type of "kubernetes\_pod\_ip" from type="ip" to type="string" and then renames it to "kube\_pod\_ip", it still didn't do anything at all, it's as if my conversion block is ignored. ```auto - convert: fields: - {from: "kubernetes_pod_ip", to: "kube_pod_ip", type: "ip"} ``` Please suggest a way out of this, we like Filebeat a lot, if only it can allow us to make such simple conversions... ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [December 18, 2023, 4:06pm UTC](https://discuss.elastic.co/t/help-with-date-time-conversion-from-filebeat/349573/2 "2023-12-18T16:06:14Z")
Hi @bobus I think there are Couple issues here..... > [@bobus](#): > > ` target: "_message"` Pretty sure that means your fields would look like `_message.timestamp` I would take a close look at the format of the output JSON and make sure you are referencing the fields correctly You can output the JSON to the console and take a look... or in elasticsearch Are you sure that is getting parsed correctly? BUT there is no `layout` parameter with convert as far as I can tell... so not sure what you got that... so I do not think convert is what you are looking for. It looks like you are trying to fix a date field.... I think for this specific use can you are looking for > **[Timestamp | Filebeat Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/processor-timestamp.html)** ---
**Author:** ![bobus](https://avatars.discourse-cdn.com/v4/letter/b/e9bcb4/32.png) [@bobus](https://discuss.elastic.co/u/bobus)\ **Post date:** [December 18, 2023, 7:54pm UTC](https://discuss.elastic.co/t/help-with-date-time-conversion-from-filebeat/349573/3 "2023-12-18T19:54:31Z")
Hello and thanks for responding, My resulting fields do look like this: ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/9/094328b228c25ba034a361086c2ef3ada92dad70.png) Yes, I'm trying to fix a date field. I want the incoming field to comply with what Elasticsearch expects: > ElasticsearchException[Elasticsearch exception [type=mapper\_parsing\_exception, reason=failed to parse field [\_message\_timestamp] of type [date] in document with id '6f8b0872-9db8-11ee-9950-0242ac1a0004'. Preview of field's value: '2023-12-18 15:16:35.781']]; nested: ElasticsearchException[Elasticsearch exception [type=illegal\_argument\_exception, **reason=failed to parse date field [2023-12-18 15:16:35.781] with format [strict\_date\_optional\_time||epoch\_millis]]];** nested: ElasticsearchException[Elasticsearch exception [type=date\_time\_parse\_exception, reason=Failed to parse with all enclosed parsers]]; **strict\_date\_optional\_time** has a "T" between date and time. My data doesn't. If "convert" is not the right path, please tell me what is. ---
**Author:** ![bobus](https://avatars.discourse-cdn.com/v4/letter/b/e9bcb4/32.png) [@bobus](https://discuss.elastic.co/u/bobus)\ **Post date:** [December 18, 2023, 8:34pm UTC](https://discuss.elastic.co/t/help-with-date-time-conversion-from-filebeat/349573/4 "2023-12-18T20:34:29Z")
Oh, and, how can I output the JSON that's sent out to the console? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [December 18, 2023, 8:49pm UTC](https://discuss.elastic.co/t/help-with-date-time-conversion-from-filebeat/349573/5 "2023-12-18T20:49:31Z")
> [@bobus](#): > > Oh, and, how can I output the JSON that's sent out to the console? Apologies I did not look closer, you are shipping from containers... Did you try what I suggested from [here](https://www.elastic.co/guide/en/beats/filebeat/current/processor-timestamp.html) ```auto processors: - timestamp: field: _message_timestamp target_field: _message_timestamp layouts: - '2006-01-02 15:04:05.999' test: - '2023-12-18 15:16:35.781' ``` ---
**Author:** ![bobus](https://avatars.discourse-cdn.com/v4/letter/b/e9bcb4/32.png) [@bobus](https://discuss.elastic.co/u/bobus)\ **Post date:** [December 19, 2023, 11:12am UTC](https://discuss.elastic.co/t/help-with-date-time-conversion-from-filebeat/349573/6 "2023-12-19T11:12:40Z")
Thanks a lot, what you suggested resolved that problem with \_message\_timestamp, yet now I have a similar problem with \_message\_ts: > ElasticsearchException[Elasticsearch exception [type=mapper\_parsing\_exception, reason=failed to parse field [\_message\_ts] of type [date] in document with id '671e3b21-9e54-11ee-9950-0242ac1a0004'. **Preview of field's value: '1.702979582618132E9'** ]]; nested: ElasticsearchException[Elasticsearch exception [type=illegal\_argument\_exception, reason=failed to parse date field [1.702979582618132E9] with format **[strict\_date\_optional\_time||epoch\_millis]]**]; nested: ElasticsearchException[Elasticsearch exception [type=date\_time\_parse\_exception, reason=Failed to parse with all enclosed parsers]]; So I applied the exact same fix as you suggested, using a 2nd timestamp processor: ```auto - timestamp: field: _message_ts target_field: _message_ts layouts: - UNIX ``` ...since "UNIX" is a standard format along with UNIX\_MS, but the error persists. It seems to me that, if I added a 3rd candidate format, using epoch\_second, I'd be able to parse this field. But where are these alternative-allowed-formats defined? **Another important question** : does this mean that the log entry does not get into the database? Or does it get in, but not fully parsed? When I search that particular stream for all entries that contain the \_message\_ts field with "_exists_: \_message\_ts", I get zero results, so it seems all failed log entries are rejected, which is a no-no for us. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [December 19, 2023, 4:30pm UTC](https://discuss.elastic.co/t/help-with-date-time-conversion-from-filebeat/349573/7 "2023-12-19T16:30:01Z")
> [@bobus](#): > > **Another important question** : does this mean that the log entry does not get into the database? Or does it get in, but not fully parsed? Yes when there is a mapping parsing exception... the document is dropped. Are you using the default template or did you create your own. What datastream is being written to.. I can not tell from the error logs. > [@bobus](#): > > [type=illegal\_argument\_exception, reason=failed to parse date field [1.702979582618132E9] with format **[strict\_date\_optional\_time||epoch\_millis]]**]; > [@bobus](#): > > 1.702979582618132E9 That is not a valid long so that is why it is not being read.. it looks something before is converting it to scientific notation. Did you manually set `_message_ts` to type date? ---
**Author:** ![bobus](https://avatars.discourse-cdn.com/v4/letter/b/e9bcb4/32.png) [@bobus](https://discuss.elastic.co/u/bobus)\ **Post date:** [December 19, 2023, 5:43pm UTC](https://discuss.elastic.co/t/help-with-date-time-conversion-from-filebeat/349573/8 "2023-12-19T17:43:03Z")
First of all thanks for continuing to help! The data stream is called "integration\_stream". What "default template" are you referring to? I don't know how the scientific notation came into play here. In the index mapping, the `_mapping_ts` field is marked as "date". I did not manually set it. Interestingly, in yesterday's index from the same index set the mapping is "float". Here is my current processor setup. As you can tell I'm getting a bit desperate with the `_message_ts` field. ``` filebeat.inputs: - type: container paths: - /var/log/containers/*.log processors: - add_kubernetes_metadata: host: ${NODE_NAME} matchers: - logs_path: logs_path: "/var/log/containers/" - decode_json_fields: fields: ["message"] target: "_message" overwrite_keys: true add_error_key: true - timestamp: field: _message_ts target_field: _message_ts layouts: - 'UNIX' - UNIX - '1123456789.123456' - 1123456789.123456 test: - 1702912588.904218 - timestamp: field: _message_timestamp target_field: _message_timestamp layouts: - '2006-01-02 15:04:05.999' test: - '2023-12-18 15:16:35.781' ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [December 19, 2023, 9:47pm UTC](https://discuss.elastic.co/t/help-with-date-time-conversion-from-filebeat/349573/9 "2023-12-19T21:47:23Z")
> [@bobus](#): > > In the index mapping, the `_mapping_ts` field is marked as "date". I did not manually set it. Interestingly, in yesterday's index from the same index set the mapping is "float". So, you are not defining a mapping. This is a source of many of your issues... A mapping is a "schema" [Index Templates](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-templates.html) and [Mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) If you do not provide a mapping ... elastic will "Guess" at the data type from the first value it sees... so if it sees something that looks like a float it picks a float if it look For any real or production use case we _ **always** _ recommend defining a template / mapping. > [@bobus](#): > > What "default template" are you referring to? So if you do not change the index name the data from filebeeat will be written into a filebeat data stream, which will come with a robust template You did not share the output section of your config so I am just learning this now. > [@bobus](#): > > ```auto > - timestamp: > field: _message_ts > target_field: _message_ts > layouts: > - 'UNIX' > - UNIX > - '1123456789.123456'