# Convert iso 8601 timestamp to UNIX\_MS
**URL:** https://discuss.elastic.co/t/convert-iso-8601-timestamp-to-unix-ms/306693
**Category:** Logstash
**Tags:** docker
**Created:** [June 8, 2022, 12:17pm UTC](https://discuss.elastic.co/t/convert-iso-8601-timestamp-to-unix-ms/306693 "2022-06-08T12:17:40Z")
**Posts on this page:** 17
**Page:** 1
### Author:  [@The-morpho](https://discuss.elastic.co/u/The-morpho)
#### Post date: [June 8, 2022, 12:17pm UTC](https://discuss.elastic.co/t/convert-iso-8601-timestamp-to-unix-ms/306693/1 "2022-06-08T12:17:40Z")
Hello,
I have json logs wich have a field timestamp in iso 8601 format I want to convert it to UNIX\_MS so I can calculate the response time between step 1 and 2, 1 and 3 for each mid.
How this can be done ?
This is an example of my log files.
```auto
{“log_level”:“INFO”,“timestamp”:“2021-12-22T11:49:06.124890Z”,“event_type”:“step1”,“mid”:“96712abc”}
{“log_level”:“INFO”,“timestamp”:“2021-12-22T11:49:07.124897Z”,“event_type”:“step2”,“mid”:“96712abc”}
{“log_level”:“INFO”,“timestamp”:“2021-12-22T11:49:07.124899Z”,“event_type”:“step3”,“mid”:“96712abc”}
{“log_level”:“INFO”,“timestamp”:“2021-12-22T11:49:08.124900Z”,“event_type”:“step1”,“mid”:“875dbca”}
```
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [June 8, 2022, 5:32pm UTC](https://discuss.elastic.co/t/convert-iso-8601-timestamp-to-unix-ms/306693/2 "2022-06-08T17:32:14Z")
Is that 3 lines or 4?
---
### Author:  [@The-morpho](https://discuss.elastic.co/u/The-morpho)
#### Post date: [June 8, 2022, 5:41pm UTC](https://discuss.elastic.co/t/convert-iso-8601-timestamp-to-unix-ms/306693/3 "2022-06-08T17:41:41Z")
4 lines, sorry It was a typing error
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [June 8, 2022, 6:29pm UTC](https://discuss.elastic.co/t/convert-iso-8601-timestamp-to-unix-ms/306693/4 "2022-06-08T18:29:06Z")
You want something very much like [example 1](https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html#plugins-filters-aggregate-example1) in the aggregate documentation.
Although the logstash Timestamp object supports nanosecond precision, the date filter does not. However, the json filter does, so we have to rename the timestamp field to @timestamp to get the json filter to use it to set [@timestamp].
```
mutate { gsub => ["message", "“", '"', "message", "”", '"'] }
mutate { gsub => ["message", "timestamp", "@timestamp"] }
json { source => "message" remove_field => ["message"] }
if [event_type] == "step1" {
aggregate {
task_id => "%{mid}"
code => 'map["step1Time"] = event.get("@timestamp").to_f'
map_action => "create"
}
}
if [event_type] == "step2" {
aggregate {
task_id => "%{mid}"
code => 'map["step2Time"] = event.get("@timestamp").to_f'
map_action => "update"
}
}
if [event_type] == "step3" {
aggregate {
task_id => "%{mid}"
code => '
map["step3Time"] = event.get("@timestamp").to_f
event.set("delta12", map["step2Time"] - map["step1Time"])
event.set("delta13", map["step3Time"] - map["step1Time"])
'
map_action => "update"
end_of_task => true
timeout => 120
}
}
```
Make sure you understand the requirements for pipeline.workers and pipeline.ordered document for the aggregate filter.
---
### Author:  [@The-morpho](https://discuss.elastic.co/u/The-morpho)
#### Post date: [June 9, 2022, 11:52am UTC](https://discuss.elastic.co/t/convert-iso-8601-timestamp-to-unix-ms/306693/5 "2022-06-09T11:52:47Z")
Thank you so much Badger, can you please explain this code, I found in the documentation that gsub filter match string field and replace them with the last value . Is the message will be removed ?
And in the aggregation we have
`event.get("@timestamp").to_f'`
Is the "to\_f" convert the timstamp to millisecond ?
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [June 9, 2022, 4:28pm UTC](https://discuss.elastic.co/t/convert-iso-8601-timestamp-to-unix-ms/306693/6 "2022-06-09T16:28:25Z")
The [message] field is modified by the gsub, not removed. The curly quotes are changed to straight quotes, and the string timestamp is replaced with @timestamp.
The .to\_f converts the LogStash::Timestamp object to a floating point number. The source JSON has microsecond precision so the floating point number will have microsecond precision. But, as always with floating point numbers, the accuracy of the digital representation is not exact. The difference between 06.124890 and 07.124897 may not be 1.000007, it might be 1.00000699827
---
### Author:  [@The-morpho](https://discuss.elastic.co/u/The-morpho)
#### Post date: [June 13, 2022, 2:19pm UTC](https://discuss.elastic.co/t/convert-iso-8601-timestamp-to-unix-ms/306693/7 "2022-06-13T14:19:53Z")
Thanks sir, I tried this on grok debugger from dev tools but it didn't work, I got this error:
`
> Unable to find pattern [mid] in Grok's pattern dictionary
`
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [June 13, 2022, 4:43pm UTC](https://discuss.elastic.co/t/convert-iso-8601-timestamp-to-unix-ms/306693/8 "2022-06-13T16:43:29Z")
Not sure why you would be using a grok debugger. There is no grok filter in the configuration I suggested.
---
### Author:  [@The-morpho](https://discuss.elastic.co/u/The-morpho)
#### Post date: [June 13, 2022, 5:04pm UTC](https://discuss.elastic.co/t/convert-iso-8601-timestamp-to-unix-ms/306693/9 "2022-06-13T17:04:09Z")
> [@Badger](#):
>
> Not sure why you would be using a grok debugger. There is no grok filter in the configuration I suggested.
Yes yes, you're right there is no need for the grok debugger, but I add the configuration to logstash.conf and nothing is done, no field is added
---
### Author:  [@The-morpho](https://discuss.elastic.co/u/The-morpho)
#### Post date: [June 14, 2022, 10:23am UTC](https://discuss.elastic.co/t/convert-iso-8601-timestamp-to-unix-ms/306693/11 "2022-06-14T10:23:21Z")
Hello Badger please can you tell me what is the purpose of using this line:
```auto
json { source => "message" remove_field => ["message"] }
```
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [June 14, 2022, 3:47pm UTC](https://discuss.elastic.co/t/convert-iso-8601-timestamp-to-unix-ms/306693/12 "2022-06-14T15:47:57Z")
Your [message] field contains JSON. That will parse it so that you have [log\_level], [timestamp], [event\_type], and [mid] fields.
If it is successfully parsed then the [message] field will be deleted, so that you do not have duplicate data. If the parsing fails then the [messsage] field will remain so that you can see what is wrong with it.
---
### Author:  [@The-morpho](https://discuss.elastic.co/u/The-morpho)
#### Post date: [June 14, 2022, 4:32pm UTC](https://discuss.elastic.co/t/convert-iso-8601-timestamp-to-unix-ms/306693/13 "2022-06-14T16:32:27Z")
Thanks you Badger for your help.
Do I have to set anything apart from the logstash configuration to run the filter?
Because I copied it as it is and it didn't work, also there is no error message in the logs of the logstash image on docker.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [June 14, 2022, 4:41pm UTC](https://discuss.elastic.co/t/convert-iso-8601-timestamp-to-unix-ms/306693/14 "2022-06-14T16:41:57Z")
> [@The-morpho](#):
>
> Do I have to set anything apart from the logstash configuration to run the filter?
I don't think so.
---
### Author:  [@The-morpho](https://discuss.elastic.co/u/The-morpho)
#### Post date: [June 14, 2022, 4:57pm UTC](https://discuss.elastic.co/t/convert-iso-8601-timestamp-to-unix-ms/306693/15 "2022-06-14T16:57:18Z")
> [@Badger](#):
>
> Make sure you understand the requirements for pipeline.workers and pipeline.ordered document for the aggregate filter.
Is there any documentation for pipeline.workers and pipeline.ordered document ?
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [June 14, 2022, 5:10pm UTC](https://discuss.elastic.co/t/convert-iso-8601-timestamp-to-unix-ms/306693/16 "2022-06-14T17:10:58Z")
They are documented [here](https://www.elastic.co/guide/en/logstash/current/logstash-settings-file.html). As with most pipeline setting they can be set globally or per-pipeline.
---
### Author:  [@The-morpho](https://discuss.elastic.co/u/The-morpho)
#### Post date: [June 29, 2022, 9:38am UTC](https://discuss.elastic.co/t/convert-iso-8601-timestamp-to-unix-ms/306693/17 "2022-06-29T09:38:08Z")
Thanks sir, can You please check this post about an error
> [@Aggregate exception occurred {:ERROR=\>#\](https://discuss.elastic.co/t/aggregate-exception-occurred-error-nomethoderror-undefined-method-for-nil-nilclass/308435):
>
> Hello, I'm new with elastic, I'm working with logs, I want to process from them a new fields based on aggregation. This the my configuration file input { beats { port =\> 5044 } } filter { json { source =\> "message" remove\_field =\> ["message"] } mutate { gsub =\> ["message", "timestamp", "@timestamp"] } if [event\_type] == "request\_inc" { aggregate { task\_id =\> "%{msg\_uuid}" code =\> 'map["step1Time"] = event.get("@timestamp").to\_f' …
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 27, 2022, 9:38am UTC](https://discuss.elastic.co/t/convert-iso-8601-timestamp-to-unix-ms/306693/18 "2022-07-27T09:38:33Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.