# @timestamp long vs. string format

**URL:** <https://discuss.elastic.co/t/timestamp-long-vs-string-format/345663>\
**Category:** Logstash\
**Created:** [October 24, 2023, 7:46pm UTC](https://discuss.elastic.co/t/timestamp-long-vs-string-format/345663 "2023-10-24T19:46:39Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![tlacuache](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tlacuache/32/39003_2.png) [@tlacuache](https://discuss.elastic.co/u/tlacuache)\
**Post date:** [October 24, 2023, 7:46pm UTC](https://discuss.elastic.co/t/timestamp-long-vs-string-format/345663/1 "2023-10-24T19:46:39Z")

</div>

I'm working on a project that has a few different components that both write documents to the same indices. One of these components is [Arkime](https://github.com/arkime/arkime), which writes its documents' [`@timestamp` `date`](https://github.com/arkime/arkime/blob/189a70fc1ca5554731c97c48103d29941c55a6ea/db/db.pl#L1322) as [UNIX milliseconds](https://github.com/arkime/arkime/blob/189a70fc1ca5554731c97c48103d29941c55a6ea/capture/db.c#L740-L742) value. This results in documents with a `@timestamp` field that looks like this:

```auto
{
  "filter": {
    "_id": "231024-SQXv7KdQjhJLAIjUhQxu7cPB"
  },
  "range": [
    0,
    1698170382
  ],
  "results": [
    {
      "_id": "231024-SQXv7KdQjhJLAIjUhQxu7cPB",
      "_index": "arkime_sessions3-231024",
      "_score": 0,
      "_source": {
        "@timestamp": 1698170342616,
        "client": {
          "bytes": 0
        },
...

```

And I can verify the mapping in the index is of type `date`:

```auto
{
  "arkime_sessions3-231024" : {
    "mappings" : {
      "@timestamp" : {
        "full_name" : "@timestamp",
        "mapping" : {
          "@timestamp" : {
            "type" : "date"
          }
        }
      }
    }
  }
}

```

Another component of my project generate logs that go through LogStash for some enrichment. I'm using the [`date` filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html) to populate the `@timestamp` field from another field's value. My project intends to support both Elasticsearch and OpenSearch backends. When using OpenSearch as a backend, I have no problem. The document is indexed, and the resulting document looks like this:

```auto
{
  "filter": {
    "_id": "231024-aHeDKjnEq5y9nn9Hj-e-EQ"
  },
  "range": [
    0,
    1698170294
  ],
  "results": [
    {
      "_id": "231024-aHeDKjnEq5y9nn9Hj-e-EQ",
      "_index": "arkime_sessions3-231024",
      "_score": 0,
      "_source": {
        "@timestamp": "2023-10-24T17:54:26.168Z",

```

From what I understand the formatting of the `@timestamp` field is because LogStash requires that field to be of type `LogStash::TimeStamp`, which is formatted according to ISO8601.

However, when I attempt to use Elasticsearch as the backend, indexing any document with LogStash results in an error that looks like this:

```auto
...
"status": 400,
"error": {
  "type": "document_parsing_exception",
  "reason": "[1:390] failed to parse field [@timestamp] of type [long] in document with id '231024--qTiw48vzayoA1Jvertxdg'. Preview of field's value: '2023-10-24T19:54:50.495Z'",
  "caused_by": {
    "type": "illegal_argument_exception",
    "reason": "For input string: \"2023-10-24T19:54:50.495Z\""
  }
}
...

```

I've tried converting the `@timestamp` field to a `long` value, setting it with a `mutate` filter, and doing it directly in embedded ruby, but all of those just result in errors in LogStash saying that it expects `@timestamp` to be of type `LogStash::TimeStamp`.

So my questions boil down to this:

1. Is there some way for me to force LogStash to allow my documents' `@timestamp` field to be a UNIX MS time? Other `date` fields besides `@timestamp` work fine.
2. If not, is there a way for me to make Elasticsearch allow both the UNIX MS and ISO8601 formats? Honestly I don't see where the `long` definition for `@timestamp` is coming from in the first place.
3. I know this is likely out of scope for this forum, but why does OpenSearch seem fine with the situation while Elasticsearch rejects it?

Warmly,

-SG

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 24, 2023, 7:46pm UTC](https://discuss.elastic.co/t/timestamp-long-vs-string-format/345663/2 "2023-10-24T19:46:39Z")

</div>

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [October 24, 2023, 8:57pm UTC](https://discuss.elastic.co/t/timestamp-long-vs-string-format/345663/3 "2023-10-24T20:57:35Z")

</div>

What is the mapping in both your indices on Elasticsearch and Opensearch?

> [1:390] failed to parse field [@timestamp] of type [long] in document with id '231024--qTiw48vzayoA1Jvertxdg'. Preview of field's value: '2023-10-24T19:54:50.495Z'"

This means that your `@timestamp` field on this indice is mapped as a `long`, not a `date`, so it will fail when you try to index a date string.

It works when you index an epoch time because it interprets the value as a long, not a date, so you will not be able to use this field in this indice as you would use a date field.

You should map the `@timestamp` as a `date` in both indices, it will accept both the date string and epoch time as the default format for the `date` type is `strict_date_optional_time||epoch_millis`.

From what you shared you have two indices with different mappings for the `@timestamp` field.

---

<div class="post-metadata">

**Author:** ![tlacuache](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tlacuache/32/39003_2.png) [@tlacuache](https://discuss.elastic.co/u/tlacuache)\
**Post date:** [October 24, 2023, 9:07pm UTC](https://discuss.elastic.co/t/timestamp-long-vs-string-format/345663/4 "2023-10-24T21:07:44Z")

</div>

I only have one index, and, as you can see from my original post (see where I said "And I can verify the mapping"...), the mapping is `date` for `@timestamp`. That's what's confusing to me about this: I only have the one index, and when I query the mapping of that field it comes back as `date`. I don't see `long` anywhere except in that error message.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [October 24, 2023, 9:19pm UTC](https://discuss.elastic.co/t/timestamp-long-vs-string-format/345663/5 "2023-10-24T21:19:51Z")

</div>

> [@tlacuache](#):
>
> the mapping is `date` for `@timestamp`.

According to this log, the mapping for the `@timestamp` field on this specific index is not `date`, it is `long`.

> [1:390] failed to parse field [@timestamp] of type [long] in document with id '231024--qTiw48vzayoA1Jvertxdg'. Preview of field's value: '2023-10-24T19:54:50.495Z'"

Can you share the full error log you are getting? The log is pretty clear already, it is a mapping error, but the full error from Logstash will have the index name and more information about the mapping.

> [@tlacuache](#):
>
> I only have one index

You mentioned that you are using both Elasticsearch and Opensearch and that this works on Opensearch, so I'm assuming that you have at least 2 indices.

Can you provide more context about it? Because the error you shared clearly shows that you have an indice where the mapping for the `@timestamp` field is of the type `long`.

You can replicate this error running the following requests in Dev Tools:

```auto
PUT discuss-sample

PUT /discuss-sample/_mapping
{
  "properties": {
    "@timestamp": {
      "type": "long"
    },
    "message": { "type": "text"}
  }
}

POST discuss-sample/_doc/
{
  "@timestamp": "2023-10-24T17:54:26.168Z",
  "message": "sample message date string"
}

```

This will result in the following mapping error:

```auto
{
  "error": {
    "root_cause": [
      {
        "type": "document_parsing_exception",
        "reason": "[2:17] failed to parse field [@timestamp] of type [long] in document with id 'JtaOY4sBIqOb1VE3RFRd'. Preview of field's value: '2023-10-24T17:54:26.168Z'"
      }
    ],
    "type": "document_parsing_exception",
    "reason": "[2:17] failed to parse field [@timestamp] of type [long] in document with id 'JtaOY4sBIqOb1VE3RFRd'. Preview of field's value: '2023-10-24T17:54:26.168Z'",
    "caused_by": {
      "type": "illegal_argument_exception",
      "reason": "For input string: \"2023-10-24T17:54:26.168Z\""
    }
  },
  "status": 400
}

```

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [October 24, 2023, 9:26pm UTC](https://discuss.elastic.co/t/timestamp-long-vs-string-format/345663/6 "2023-10-24T21:26:32Z")

</div>

> [@tlacuache](#):
>
> However, when I attempt to use Elasticsearch as the backend, indexing any document with LogStash results in an error that looks like this

Also, can you share your Logstash output in this case with the index name and the full log error you are getting.

---

<div class="post-metadata">

**Author:** ![tlacuache](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tlacuache/32/39003_2.png) [@tlacuache](https://discuss.elastic.co/u/tlacuache)\
**Post date:** [October 24, 2023, 9:29pm UTC](https://discuss.elastic.co/t/timestamp-long-vs-string-format/345663/7 "2023-10-24T21:29:39Z")

</div>

After reading your comments I realized something and found I had a problem with my configuration (a rogue data source I wasn't tracking writing documents before the template could be set).

In other words, the whole thing is pebkac.

I'm going to delete this thread as it's noise. I apologize for wasting your time.

---

<div class="post-metadata">

**Author:** ![tlacuache](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tlacuache/32/39003_2.png) [@tlacuache](https://discuss.elastic.co/u/tlacuache)\
**Post date:** [October 24, 2023, 9:37pm UTC](https://discuss.elastic.co/t/timestamp-long-vs-string-format/345663/8 "2023-10-24T21:37:43Z")

</div>

Apparently I can't delete the thread, but I'll mark it as "solution." Again, thanks for your time and thoughts despite the problem being user error.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 21, 2023, 9:38pm UTC](https://discuss.elastic.co/t/timestamp-long-vs-string-format/345663/9 "2023-11-21T21:38:21Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.