How to automatically set the server date/time when adding data?

Elastic Stack Elasticsearch
1

Hi there,

In old ES versions, I was using the _timestamp field but it's been removed and it's now recommended to set the date on the client side. However, I'm surprised to not find any easy way to continue to set it on the server side. Setting the date on the client side is risky for our use case since we don't control the clients (they can be anywhere in the world) and they can put wrong date or in the wrong timezone. Our use case is to record pings from clients and I'd much prefer to set the date using the ES server date/time.

Is there an easy way to do this?

Thanks a lot
-Vincent

2

Bonjour Vincent! :wink:

I believe you can do something like this using an ingest pipeline with a Set processor.

And you can make the pipeline a default pipeline for your index. (See Ingest pipelines | Elasticsearch Guide [7.17] | Elastic)

Full example (from this example Ingest pipelines | Elasticsearch Guide [7.17] | Elastic):

PUT _ingest/pipeline/set-timestamp
{
  "description": "sets the timestamp",
  "processors": [
    {
      "set": {
        "field": "timestamp",
        "value": "{{{_ingest.timestamp}}}"
      }
    }
  ]
}
DELETE test
PUT test
{
  "settings": {
    "index.default_pipeline": "set-timestamp"
  },
  "mappings": {
    "properties": {
      "timestamp": {
        "type": "date"
      }
    }
  }
}
PUT test/_doc/1
{
  "foo": "bar"
}
GET test/_doc/1

Gives:

{
  "_index" : "test",
  "_type" : "_doc",
  "_id" : "1",
  "_version" : 1,
  "_seq_no" : 0,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "foo" : "bar",
    "timestamp" : "2022-02-09T10:47:34.880561Z"
  }
}
3

hey, hi David! :slight_smile:

Thanks a lot for your answer.

I should have mentioned that I saw this answer in some answer on stackoverflow but I thought it looked a bit complex, hence why I asked the question here. So I gather that there's no simpler way and I'll go in this direction, thanks again!

PS: The concept of pipelines is pretty cool!

4

@dadoonet Sorry, another question related to this. Do you know if this ingest pipeline will overwrite any value set in the timestamp field from your example? Is it possible to say to replace the field value only if it's empty or not set?

Thanks

5

To be more precise, I'd like to set 2 dates:

  • pingDate: always set it with an ingest pipeline to be the current server date
  • firstPingDate: I do some query to find the first ping entry's firstPingDate for a given instance id. If it exists, I set it to that value. If it doesn't exist, I'd like to set it to the current server date. I'd like it to be the same value as the pingDate too.

So 3 questions:

  1. Do I need 2 pipelines?
  2. How can I set firstPingDate in a pipeline, only if not set already?
  3. How can I be sure to use the same value as the pingDate provided by the ES server?

Lots of questions, sorry...

Thanks!

6

Actually I'm using the wrong terminology. I need 2 processors I think.

7

Found the answer with the override config param in Set processor | Elasticsearch Guide [7.17] | Elastic

1 Like
8

Af far as you are using Update API, override parameter or if paramter in set processors can detect whether there is existing value or not.

But if you use Index API, both paramters does not work.

9

Seems I coud use the copy_from config param from Set processor | Elasticsearch Guide [7.17] | Elastic :slight_smile:

10

@dadoonet hmm I can't find aJava API to use copy_from, it seems to be missing. This is what I'm trying:

        client.ingest().putPipeline(b0 -> b0
            .id("set-timestamp")
            .description("Set pingDate to be the server date/time and fill firstPingDate if empty")
            .processors(b1 -> b1
                .set(b2 -> b2
                    .field(PROPERTY_PING_DATE)
                    .value(JsonData.of("{{{_ingest.timestamp}}}")))
                .set(b3 -> b3
                    .override(false)
                    .field(PROPERTY_FIRST_PING_DATE)
                    .copy_from(PROPERTY_PING_DATE)
                )));

BTW the second set() isn't correct syntactically. How do I setup 2 "set" processors using the java API?

Thx!

11

For the second question re multiple processors, it seems the API is not finished but I've found a workaround with:

        Processor processor1 = Processor.of(b0 -> b0
            .set(b1 -> b1
                .field(PROPERTY_PING_DATE)
                .value(JsonData.of("{{{_ingest.timestamp}}}"))));
        Processor processor2 = Processor.of(b0 -> b0
            .set(b3 -> b3
                .override(false)
                .field(PROPERTY_FIRST_PING_DATE)
                .copy_from(PROPERTY_PING_DATE)));
        client.ingest().putPipeline(b0 -> b0
            .id("set-timestamp")
            .description("Set pingDate to be the server date/time and fill firstPingDate if empty")
            .processors(processor1, processor2);

Note that copy_from() is not working as it doesn't exist. So I still don't know how to achieve that part.

Thx

12

Bringing in @swallez as this part is the one he is working on.

13

That said, why not using value in that case?

Like "value": "{{{timestamp}}}"

14

Yes that works fine, as a workaround (I guess it would be cleaner to use copy_from since it's meant for that) :slight_smile:

Thanks!

For future readers, this is what I got to:

        Processor processor1 = Processor.of(b0 -> b0
            .set(b1 -> b1
                .field(PROPERTY_PING_DATE)
                .value(JsonData.of("{{{_ingest.timestamp}}}"))));
        Processor processor2 = Processor.of(b0 -> b0
            .set(b3 -> b3
                .override(false)
                .field(PROPERTY_FIRST_PING_DATE)
                .value(JsonData.of(String.format("{{{%s}}}", PROPERTY_PING_DATE)))));
        client.ingest().putPipeline(b0 -> b0
            .id(PIPELINE_NAME)
            .description("Set pingDate to be the server date/time and fill firstPingDate if empty")
            .processors(processor1, processor2));
15

Thank @dadoonet. It's actually missing in the API specification and I opened an issue to fix this.

1 Like
16

To me copy_from is helpful when you want to copy a full structure of fields, like an object.
For a single field, I think that value is the way to go.

17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.