# Can't delete/change data in @timestamp field yielded by ingest processor
**URL:** https://discuss.elastic.co/t/cant-delete-change-data-in-timestamp-field-yielded-by-ingest-processor/294497
**Category:** Elasticsearch
**Tags:** painless
**Created:** [January 16, 2022, 10:32am UTC](https://discuss.elastic.co/t/cant-delete-change-data-in-timestamp-field-yielded-by-ingest-processor/294497 "2022-01-16T10:32:27Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@SomeAkk](https://discuss.elastic.co/u/SomeAkk)
#### Post date: [January 16, 2022, 10:32am UTC](https://discuss.elastic.co/t/cant-delete-change-data-in-timestamp-field-yielded-by-ingest-processor/294497/1 "2022-01-16T10:32:27Z")
Hi! Today i add to my document `@timestamp` field (before this value was not in the document, and document no any date data.) with date data using ingest set processor:
```auto
{
"processors": [
{
"set": {
"description": "Add timestamp to document from ingest timestamp as '@timestamp'",
"field": "@timestamp",
"value": "{{{_ingest.timestamp}}}"
}
}
]
}
```
Then i tried to update\_by\_query that field.
After that i want to change data value of `@timestamp` like this:
```auto
POST materials_test/_update_by_query?conflicts=proceed
{
"script" : "ctx._source['@timestamp'] = ZonedDateTime.parse('2022-01-01T00:00:00+03:00')",
"query" : {
"term": {
"_id": "96091631"
}
}
}
```
**And there is no any effect** - @timestamp not changed.
Then i thing that i can delete that field also using update\_by\_query:
```auto
POST materials_test/_update_by_query?conflicts=proceed
{
"script" : {
"lang": "painless",
"source": "ctx._source.remove('@timestamp')"
},
"query" : {
"term": {
"_id": "96091631"
}
}
}
```
**And there is no any effect too** - @timestamp stay in the document.
UPDATE.
- also try: `POST materials_test/_forcemerge?only_expunge_deletes=true`
- also try: `POST materials_test/_flush`
- also try: `POST _cache/clear`
Why it so? Fields protected?
Name of my timestamp field must not starts with "@"?
How i can now delete this field from document?
> Environment:
>
> - Elasticsearch version: 7.10
> Index:
>
> ```auto
> {
> "materials_test" : {
> "settings" : {
> "index" : {
> "routing" : {
> "allocation" : {
> "include" : {
> "_tier_preference" : "data_content"
> }
> }
> },
> "number_of_shards" : "1",
> "provided_name" : "materials_test",
> "default_pipeline" : "add_timestamp",
> "creation_date" : "1611318533803",
> "number_of_replicas" : "0",
> "uuid" : "YUsr8w9aQDGg9YfOHBUbGQ",
> "version" : {
> "created" : "7100299"
> }
> }
> }
> }
> }
>
> ```
---
### Author:  [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)
#### Post date: [January 18, 2022, 4:17pm UTC](https://discuss.elastic.co/t/cant-delete-change-data-in-timestamp-field-yielded-by-ingest-processor/294497/2 "2022-01-18T16:17:57Z")
> [@SomeAkk](#):
>
> **And there is no any effect** - `@timestamp` not changed.
> [@SomeAkk](#):
>
> **And there is no any effect too** - `@timestamp` stay in the document.
There were really no effects or the `@timestamp` field updated to the newer timestamp.
In any case, I suppose the behavior is in line with the specification of elasticsearch. Your settings show `"add_timestamp"` is stiil working as `"default_pipeline"`. The pipeline works AFTER script field performed and set `{{{_ingest.timestamp}}}` to `@timestamp` field finaly.
If you want to delete the field, change the ingest processor.
---
### Author:  [@SomeAkk](https://discuss.elastic.co/u/SomeAkk)
#### Post date: [January 18, 2022, 4:45pm UTC](https://discuss.elastic.co/t/cant-delete-change-data-in-timestamp-field-yielded-by-ingest-processor/294497/3 "2022-01-18T16:45:45Z")
@Tomo_M I am just do that steps step by step:
1. create ingest processor
2. set it as default pipeline for index
3. check that that processor start add `@tamstamp` to documents
4. then i made pause for 10-15 minutes
5. then tried to update `@tamstamp` by mine value in update\_by\_query
So you want say that at:
at 6 step that field was changed to my value
at 7 step ingest processor set its own value from `_ingest.timestamp`?
by that i didn't see any changes?
I understood you right?
Up today i thing that it works only when data added to es, and not when it updated too. Interesting, need to find another solution to set `@tamstamp` - i need set it only when add document and not when it updated.
---
### Author:  [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)
#### Post date: [January 18, 2022, 4:54pm UTC](https://discuss.elastic.co/t/cant-delete-change-data-in-timestamp-field-yielded-by-ingest-processor/294497/4 "2022-01-18T16:54:08Z")
Oh, now I know what you have been misunderstainding.
Ingest pipeline also works when documents are updated not only when the document was created. Updating is internaly the same as indexing a new document and delete the old document.
Please see the @timestamp field after updating the document, and compare with the value just before the updating. You will realize the ingest pipeline is also working on updating.
---
### Author:  [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)
#### Post date: [January 18, 2022, 5:11pm UTC](https://discuss.elastic.co/t/cant-delete-change-data-in-timestamp-field-yielded-by-ingest-processor/294497/5 "2022-01-18T17:11:53Z")
I found an interesting idea. I'm not sure that is useful today but `override` option is still usable in set processor. It looks worth trying.
>
>
> Would it make sense to support ingest pipelines as part of the update api?
>
> Inge…st pipelines can already be used in the reindex api as an alternative to scripts. There is some kind of overlap between ingest and scripts: scripts are much more flexible (and safe with painless) but for people that get familiar with ingest it may be more convenient to reuse defined pipelines. Also some of the enrichments made through ingest are not easily performed using scripts (think geoip).
>
> We can debate that ingest enrichments are really meant to be executed pre-indexing. But while we are reindexing a document, which is what the update api does, maybe it would be handy to be able to use ingest pipelines as well? Or maybe this is a bad idea at all and it would introduce yet another way to update documents other than scripts and partial document. I am on the fence on this myself, would be good to discuss this.
>
> This \[comment\](https://github.com/elastic/elasticsearch/pull/16808#issuecomment-212485004) made me think about this. Not sure ingest would help there though.
---
### Author:  [@stu](https://discuss.elastic.co/u/stu)
#### Post date: [January 18, 2022, 5:29pm UTC](https://discuss.elastic.co/t/cant-delete-change-data-in-timestamp-field-yielded-by-ingest-processor/294497/6 "2022-01-18T17:29:35Z")
Hi @SomeAkk,
> Please see the @timestamp field after updating the document, and compare with the value just before the updating. You will realize the ingest pipeline is also working on updating.
@Tomo_M is correct about the functioning of pipelines.
> i need set it only when add document and not when it updated.
One way to do this is attach an [if](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/set-processor.html) to your set processor in your pipeline.
```auto
PUT /_ingest/pipeline/materials_test_pipeline
{
"processors": [
{
"set": {
"description": "Add timestamp to document from ingest timestamp as '@timestamp'",
"field": "@timestamp",
"value": "{{{_ingest.timestamp}}}",
"if": "!ctx.containsKey('@timestamp')"
}
}
]
}
```
`"if": "!ctx.containsKey('@timestamp')"` will only set the timestamp if the document does not already have a `@timestamp` field.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 15, 2022, 5:29pm UTC](https://discuss.elastic.co/t/cant-delete-change-data-in-timestamp-field-yielded-by-ingest-processor/294497/7 "2022-02-15T17:29:45Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.