Can't delete/change data in @timestamp field yielded by ingest processor

Elastic Stack Elasticsearch
1

Hi! Today i add to my document @timestamp field (before this value was not in the document, and document no any date data.) with date data using ingest set processor:

{
  "processors": [
    {
      "set": {
        "description": "Add timestamp to document from ingest timestamp as '@timestamp'",
        "field": "@timestamp",
        "value": "{{{_ingest.timestamp}}}"
      }
    }
  ]
}

Then i tried to update_by_query that field.

After that i want to change data value of @timestamp like this:

POST materials_test/_update_by_query?conflicts=proceed
{
    "script" : "ctx._source['@timestamp'] = ZonedDateTime.parse('2022-01-01T00:00:00+03:00')",
    "query" : {
        "term": { 
          "_id": "96091631"
        }
    }
}

And there is no any effect - @timestamp not changed.

Then i thing that i can delete that field also using update_by_query:

POST materials_test/_update_by_query?conflicts=proceed
{
    "script" : {
      "lang": "painless",
      "source": "ctx._source.remove('@timestamp')"
    },
    "query" : {
        "term": { 
          "_id": "96091631"
        }
    }
}

And there is no any effect too - @timestamp stay in the document.

UPDATE.

  • also try: POST materials_test/_forcemerge?only_expunge_deletes=true
  • also try: POST materials_test/_flush
  • also try: POST _cache/clear

Why it so? Fields protected?
Name of my timestamp field must not starts with "@"?
How i can now delete this field from document?

Environment:

  • Elasticsearch version: 7.10

Index:

{
  "materials_test" : {
    "settings" : {
      "index" : {
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "number_of_shards" : "1",
        "provided_name" : "materials_test",
        "default_pipeline" : "add_timestamp",
        "creation_date" : "1611318533803",
        "number_of_replicas" : "0",
        "uuid" : "YUsr8w9aQDGg9YfOHBUbGQ",
        "version" : {
          "created" : "7100299"
        }
      }
    }
  }
}
2

There were really no effects or the @timestamp field updated to the newer timestamp.

In any case, I suppose the behavior is in line with the specification of elasticsearch. Your settings show "add_timestamp" is stiil working as "default_pipeline". The pipeline works AFTER script field performed and set {{{_ingest.timestamp}}} to @timestamp field finaly.

If you want to delete the field, change the ingest processor.

3

@Tomo_M I am just do that steps step by step:

  1. create ingest processor
  2. set it as default pipeline for index
  3. check that that processor start add @tamstamp to documents
  4. then i made pause for 10-15 minutes
  5. then tried to update @tamstamp by mine value in update_by_query

So you want say that at:
at 6 step that field was changed to my value
at 7 step ingest processor set its own value from _ingest.timestamp?
by that i didn't see any changes?

I understood you right?

Up today i thing that it works only when data added to es, and not when it updated too. Interesting, need to find another solution to set @tamstamp - i need set it only when add document and not when it updated.

4

Oh, now I know what you have been misunderstainding.

Ingest pipeline also works when documents are updated not only when the document was created. Updating is internaly the same as indexing a new document and delete the old document.

Please see the @timestamp field after updating the document, and compare with the value just before the updating. You will realize the ingest pipeline is also working on updating.

1 Like
5

I found an interesting idea. I'm not sure that is useful today but override option is still usable in set processor. It looks worth trying.

1 Like
6

Hi @SomeAkk,

Please see the @timestamp field after updating the document, and compare with the value just before the updating. You will realize the ingest pipeline is also working on updating.

@Tomo_M is correct about the functioning of pipelines.

i need set it only when add document and not when it updated.

One way to do this is attach an if to your set processor in your pipeline.

PUT /_ingest/pipeline/materials_test_pipeline
{
  "processors": [
    {
      "set": {
        "description": "Add timestamp to document from ingest timestamp as '@timestamp'",
        "field": "@timestamp",
        "value": "{{{_ingest.timestamp}}}",
        "if": "!ctx.containsKey('@timestamp')"
      }
    }
  ]
}

"if": "!ctx.containsKey('@timestamp')" will only set the timestamp if the document does not already have a @timestamp field.

2 Likes
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.