# Logstash 8 @timestamp field format was changed to microseconds percision

**URL:** <https://discuss.elastic.co/t/logstash-8-timestamp-field-format-was-changed-to-microseconds-percision/310852>\
**Category:** Logstash\
**Created:** [July 28, 2022, 10:44am UTC](https://discuss.elastic.co/t/logstash-8-timestamp-field-format-was-changed-to-microseconds-percision/310852 "2022-07-28T10:44:17Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ofir\_Edi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ofir_edi/32/78181_2.png) [@Ofir\_Edi](https://discuss.elastic.co/u/Ofir_Edi)\
**Post date:** [July 28, 2022, 10:44am UTC](https://discuss.elastic.co/t/logstash-8-timestamp-field-format-was-changed-to-microseconds-percision/310852/1 "2022-07-28T10:44:17Z")

</div>

Hi,  
I have been using Logstash for a while now and when upgrading to version 8 I can see the @timestamp field format was changed from milliseconds percision to microseconds percision (meaning instead of `2022-07-28T09:46:06.200Z` we are getting `2022-07-28T09:46:06.200000Z`) .  
I have many indices which map @timestamp field with milliseconds format and now i'm getting many indexing errors due to @timestamp field.

i'm able to workaround this (and change it to milliseconds percision) by overriding the field using:

```auto
mutate {
        add_field => {
            "tmptimestamp" => "%{@timestamp}"
        }
    }
    mutate {
        gsub => [
          "tmptimestamp", "\d{3}Z$", "Z"
        ]
    }
    date {
        match => ["tmptimestamp", "yyyy-MM-dd'T'HH:mm:ss.SSSZ"]
    }

```

However, this seems very ineffective and requires changing many pipelines.

Is there a way to set @timestamp format in order to get milliseconds percision in Logstash 8 as well or is there more effective solution for this?

Thanks,  
Ofir

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 28, 2022, 3:45pm UTC](https://discuss.elastic.co/t/logstash-8-timestamp-field-format-was-changed-to-microseconds-percision/310852/2 "2022-07-28T15:45:27Z")

</div>

> [@Ofir\_Edi](#):
>
> i'm getting many indexing errors due to @timestamp field

What errors do you get and what do the mappings of the fields look like?

---

<div class="post-metadata">

**Author:** ![Ofir\_Edi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ofir_edi/32/78181_2.png) [@Ofir\_Edi](https://discuss.elastic.co/u/Ofir_Edi)\
**Post date:** [July 28, 2022, 8:36pm UTC](https://discuss.elastic.co/t/logstash-8-timestamp-field-format-was-changed-to-microseconds-percision/310852/3 "2022-07-28T20:36:45Z")

</div>

Hi @Badger

mapping is like the following (only showing @timestamp part):

```auto
{
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date",
        "format": "yyyy-MM-dd'T'HH:mm:ss.SSSX"
      }
    }
  }
}

```

The error i'm getting (400 status):

```auto
{
  "error" : {
    "root_cause" : [
      {
        "type" : "mapper_parsing_exception",
        "reason" : "failed to parse field [@timestamp] of type [date] in document with id 'XXX'. Preview of field's value: '2022-07-28T08:09:38.000000Z'"
      }
    ],
    "type" : "mapper_parsing_exception",
    "reason" : "failed to parse field [@timestamp] of type [date] in document with id 'XXX'. Preview of field's value: '2022-07-28T08:09:38.000000Z'",
    "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "failed to parse date field [2022-07-28T08:09:38.000000Z] with format [yyyy-MM-dd'T'HH:mm:ss.SSSX]",
      "caused_by" : {
        "type" : "date_time_parse_exception",
        "reason" : "date_time_parse_exception: Text '2022-07-28T08:09:38.000000Z' could not be parsed at index 23"
      }
    }
  },
  "status" : 400
}

```

- I guess originally we could have just used date and let Elastic infer the format but this is already applied for many indices and current mapping can not be changed without reindexing.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 28, 2022, 8:49pm UTC](https://discuss.elastic.co/t/logstash-8-timestamp-field-format-was-changed-to-microseconds-percision/310852/4 "2022-07-28T20:49:09Z")

</div>

I do not run elasticsearch, so I cannot test this, but would [updating](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-mapping.html) the mapping on the existing index to be

```
"format": "yyyy-MM-dd'T'HH:mm:ss.SSSX||yyyy-MM-dd'T'HH:mm:ss.SSSSSSX"

```

work?

---

<div class="post-metadata">

**Author:** ![Ofir\_Edi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ofir_edi/32/78181_2.png) [@Ofir\_Edi](https://discuss.elastic.co/u/Ofir_Edi)\
**Post date:** [July 28, 2022, 9:25pm UTC](https://discuss.elastic.co/t/logstash-8-timestamp-field-format-was-changed-to-microseconds-percision/310852/5 "2022-07-28T21:25:52Z")

</div>

Unfortunately changing format is equivalent to try and change mapping on Elasticsearch and such attempt returns in conflict error:

```auto
{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "Mapper for [@timestamp] conflicts with existing mapper:\n\tCannot update parameter [format] from [yyyy-MM-dd'T'HH:mm:ss.SSSX] to [yyyy-MM-dd'T'HH:mm:ss.SSSX || yyyy-MM-dd'T'HH:mm:ss.SSSSSSX]"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "Mapper for [@timestamp] conflicts with existing mapper:\n\tCannot update parameter [format] from [yyyy-MM-dd'T'HH:mm:ss.SSSX] to [yyyy-MM-dd'T'HH:mm:ss.SSSX || yyyy-MM-dd'T'HH:mm:ss.SSSSSSX]"
  },
  "status" : 400
}

```

Also I rather avoid (if possible) changing Elasticsearch mapping and find an effective solution on Logstash side.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 28, 2022, 9:32pm UTC](https://discuss.elastic.co/t/logstash-8-timestamp-field-format-was-changed-to-microseconds-percision/310852/6 "2022-07-28T21:32:07Z")

</div>

On the logstash side the only thing I can think of would be the mutate+mutate+date that you showed.

---

<div class="post-metadata">

**Author:** ![Ofir\_Edi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ofir_edi/32/78181_2.png) [@Ofir\_Edi](https://discuss.elastic.co/u/Ofir_Edi)\
**Post date:** [July 31, 2022, 6:24am UTC](https://discuss.elastic.co/t/logstash-8-timestamp-field-format-was-changed-to-microseconds-percision/310852/7 "2022-07-31T06:24:22Z")

</div>

Thanks, I might open feature request on Github

---

<div class="post-metadata">

**Author:** ![Ofir\_Edi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ofir_edi/32/78181_2.png) [@Ofir\_Edi](https://discuss.elastic.co/u/Ofir_Edi)\
**Post date:** [August 1, 2022, 7:23am UTC](https://discuss.elastic.co/t/logstash-8-timestamp-field-format-was-changed-to-microseconds-percision/310852/8 "2022-08-01T07:23:38Z")

</div>

Opened enhancement request in case anyone wants to follow: [Configuration for @timestamp field format · Issue #14399 · elastic/logstash · GitHub](https://github.com/elastic/logstash/issues/14399)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 29, 2022, 7:24am UTC](https://discuss.elastic.co/t/logstash-8-timestamp-field-format-was-changed-to-microseconds-percision/310852/9 "2022-08-29T07:24:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.