@timestamp long vs. string format

[tlacuache]

I'm working on a project that has a few different components that both write documents to the same indices. One of these components is Arkime, which writes its documents' @timestamp date as UNIX milliseconds value. This results in documents with a @timestamp field that looks like this:

{
  "filter": {
    "_id": "231024-SQXv7KdQjhJLAIjUhQxu7cPB"
  },
  "range": [
    0,
    1698170382
  ],
  "results": [
    {
      "_id": "231024-SQXv7KdQjhJLAIjUhQxu7cPB",
      "_index": "arkime_sessions3-231024",
      "_score": 0,
      "_source": {
        "@timestamp": 1698170342616,
        "client": {
          "bytes": 0
        },
...

And I can verify the mapping in the index is of type date:

{
  "arkime_sessions3-231024" : {
    "mappings" : {
      "@timestamp" : {
        "full_name" : "@timestamp",
        "mapping" : {
          "@timestamp" : {
            "type" : "date"
          }
        }
      }
    }
  }
}

Another component of my project generate logs that go through LogStash for some enrichment. I'm using the date filter to populate the @timestamp field from another field's value. My project intends to support both Elasticsearch and OpenSearch backends. When using OpenSearch as a backend, I have no problem. The document is indexed, and the resulting document looks like this:

{
  "filter": {
    "_id": "231024-aHeDKjnEq5y9nn9Hj-e-EQ"
  },
  "range": [
    0,
    1698170294
  ],
  "results": [
    {
      "_id": "231024-aHeDKjnEq5y9nn9Hj-e-EQ",
      "_index": "arkime_sessions3-231024",
      "_score": 0,
      "_source": {
        "@timestamp": "2023-10-24T17:54:26.168Z",

From what I understand the formatting of the @timestamp field is because LogStash requires that field to be of type LogStash::TimeStamp, which is formatted according to ISO8601.

However, when I attempt to use Elasticsearch as the backend, indexing any document with LogStash results in an error that looks like this:

...
"status": 400,
"error": {
  "type": "document_parsing_exception",
  "reason": "[1:390] failed to parse field [@timestamp] of type [long] in document with id '231024--qTiw48vzayoA1Jvertxdg'. Preview of field's value: '2023-10-24T19:54:50.495Z'",
  "caused_by": {
    "type": "illegal_argument_exception",
    "reason": "For input string: \"2023-10-24T19:54:50.495Z\""
  }
}
...

I've tried converting the @timestamp field to a long value, setting it with a mutate filter, and doing it directly in embedded ruby, but all of those just result in errors in LogStash saying that it expects @timestamp to be of type LogStash::TimeStamp.

So my questions boil down to this:

1. Is there some way for me to force LogStash to allow my documents' @timestamp field to be a UNIX MS time? Other date fields besides @timestamp work fine.
2. If not, is there a way for me to make Elasticsearch allow both the UNIX MS and ISO8601 formats? Honestly I don't see where the long definition for @timestamp is coming from in the first place.
3. I know this is likely out of scope for this forum, but why does OpenSearch seem fine with the situation while Elasticsearch rejects it?

Warmly,

-SG

[system]

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns )

[leandrojmp (Leandro Pereira)]

What is the mapping in both your indices on Elasticsearch and Opensearch?

[1:390] failed to parse field [@timestamp] of type [long] in document with id '231024--qTiw48vzayoA1Jvertxdg'. Preview of field's value: '2023-10-24T19:54:50.495Z'"

This means that your @timestamp field on this indice is mapped as a long, not a date, so it will fail when you try to index a date string.

It works when you index an epoch time because it interprets the value as a long, not a date, so you will not be able to use this field in this indice as you would use a date field.

You should map the @timestamp as a date in both indices, it will accept both the date string and epoch time as the default format for the date type is strict_date_optional_time||epoch_millis.

From what you shared you have two indices with different mappings for the @timestamp field.

[tlacuache]

I only have one index, and, as you can see from my original post (see where I said "And I can verify the mapping"...), the mapping is date for @timestamp. That's what's confusing to me about this: I only have the one index, and when I query the mapping of that field it comes back as date. I don't see long anywhere except in that error message.

[leandrojmp (Leandro Pereira)]

@tlacuache

the mapping is date for @timestamp.

According to this log, the mapping for the @timestamp field on this specific index is not date, it is long.

[1:390] failed to parse field [@timestamp] of type [long] in document with id '231024--qTiw48vzayoA1Jvertxdg'. Preview of field's value: '2023-10-24T19:54:50.495Z'"

Can you share the full error log you are getting? The log is pretty clear already, it is a mapping error, but the full error from Logstash will have the index name and more information about the mapping.

@tlacuache

I only have one index

You mentioned that you are using both Elasticsearch and Opensearch and that this works on Opensearch, so I'm assuming that you have at least 2 indices.

Can you provide more context about it? Because the error you shared clearly shows that you have an indice where the mapping for the @timestamp field is of the type long.

You can replicate this error running the following requests in Dev Tools:

PUT discuss-sample

PUT /discuss-sample/_mapping
{
  "properties": {
    "@timestamp": {
      "type": "long"
    },
    "message": { "type": "text"}
  }
}

POST discuss-sample/_doc/
{
  "@timestamp": "2023-10-24T17:54:26.168Z",
  "message": "sample message date string"
}

This will result in the following mapping error:

{
  "error": {
    "root_cause": [
      {
        "type": "document_parsing_exception",
        "reason": "[2:17] failed to parse field [@timestamp] of type [long] in document with id 'JtaOY4sBIqOb1VE3RFRd'. Preview of field's value: '2023-10-24T17:54:26.168Z'"
      }
    ],
    "type": "document_parsing_exception",
    "reason": "[2:17] failed to parse field [@timestamp] of type [long] in document with id 'JtaOY4sBIqOb1VE3RFRd'. Preview of field's value: '2023-10-24T17:54:26.168Z'",
    "caused_by": {
      "type": "illegal_argument_exception",
      "reason": "For input string: \"2023-10-24T17:54:26.168Z\""
    }
  },
  "status": 400
}

[leandrojmp (Leandro Pereira)]

@tlacuache

However, when I attempt to use Elasticsearch as the backend, indexing any document with LogStash results in an error that looks like this

Also, can you share your Logstash output in this case with the index name and the full log error you are getting.

[tlacuache]

After reading your comments I realized something and found I had a problem with my configuration (a rogue data source I wasn't tracking writing documents before the template could be set).

In other words, the whole thing is pebkac.

I'm going to delete this thread as it's noise. I apologize for wasting your time.

[tlacuache]

Apparently I can't delete the thread, but I'll mark it as "solution." Again, thanks for your time and thoughts despite the problem being user error.