# Mapping - transform: only for creating new and not for updating?

**URL:** <https://discuss.elastic.co/t/mapping-transform-only-for-creating-new-and-not-for-updating/27312>\
**Category:** Elasticsearch\
**Created:** [August 13, 2015, 9:23am UTC](https://discuss.elastic.co/t/mapping-transform-only-for-creating-new-and-not-for-updating/27312 "2015-08-13T09:23:51Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sven\_Beauprez](https://avatars.discourse-cdn.com/v4/letter/s/13edae/32.png) [@Sven\_Beauprez](https://discuss.elastic.co/u/Sven_Beauprez)\
**Post date:** [August 13, 2015, 9:23am UTC](https://discuss.elastic.co/t/mapping-transform-only-for-creating-new-and-not-for-updating/27312/1 "2015-08-13T09:23:51Z")

</div>

I tested follow mapping - transform script

> "mappings": {  
> "tag": {  
> "\_timestamp": {  
> "enabled": true  
> },  
> "transform": {  
> "script": "ctx.\_source.counter = ctx.\_source.counter == null ? 1 : 2",  
> "lang": "groovy"  
> },  
> ...  
> "counter": {  
> "type": "long",  
> "index": "not\_analyzed",  
> "store" : true,  
> "doc\_values": true  
> },  
> ...

and it worked fine when indexing a new document, the `counter` field is set to `1` for each new entry added.

When I want to update/overwrite an existing entry, the script is not executed anymore. In other words, the counter is not set to `2`. Is this correct?

I would like to prevent using scripting inside the update API request (no need for the client to know this is needed), but still have some modifications in documents that are being updated in ES.

I also thought I would be able to check with `ctx.op` if a document is created or updated...

Any ideas?

regards,

Sven

---

<div class="post-metadata">

**Author:** ![Sven\_Beauprez](https://avatars.discourse-cdn.com/v4/letter/s/13edae/32.png) [@Sven\_Beauprez](https://discuss.elastic.co/u/Sven_Beauprez)\
**Post date:** [August 13, 2015, 12:42pm UTC](https://discuss.elastic.co/t/mapping-transform-only-for-creating-new-and-not-for-updating/27312/2 "2015-08-13T12:42:37Z")

</div>

Just to add some context, maybe @nik9000 or @Mark_Harwood have some ideas :

I am using the entity centric approach to store processed information in a second index based on log data, which is stored 'as is' (raw) in another index.

Data comes via logstash which uses bulk upload (another argument why scripts in update is not an option: it is not supported) towards both indexes. A unique id is used to update data in the second index, but some processing needs to be done before it can be stored (similar as session duration when talking about web logs)

I would like to avoid any external processing (ie. outside ES, such as in python) that takes data from index one, process it and store it in the second index as shown in the presentations of Mark. It seems mapping - transform should work in my simple case, but I got stuck with the above.

regards,

Sven

---

<div class="post-metadata">

**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\
**Post date:** [August 13, 2015, 2:35pm UTC](https://discuss.elastic.co/t/mapping-transform-only-for-creating-new-and-not-for-updating/27312/3 "2015-08-13T14:35:18Z")

</div>

> [@Sven\_Beauprez](#):
>
> I would like to prevent using scripting inside the update API request (no need for the client to know this is needed), but still have some modifications in documents that are being updated in ES.

That's really not the point of transforms. They are supposed to be super `copy_to`s. The point is that the data from the transform _shouldn't_ be in the `_source` and its a bug if it is.

> [@Sven\_Beauprez](#):
>
> Data comes via logstash which uses bulk upload (another argument why scripts in update is not an option: it is not supported) towards both indexes

Scripts are supported on bulk update. I dunno if logstash supports them - it should.

> [@Sven\_Beauprez](#):
>
> I would like to avoid any external processing (ie. outside ES, such as in python) that takes data from index one, process it and store it in the second index as shown in the presentations of Mark. It seems mapping - transform should work in my simple case, but I got stuck with the above.

Transform is for transforming. The update script is for updating. I think the update script is more right here.

---

<div class="post-metadata">

**Author:** ![Sven\_Beauprez](https://avatars.discourse-cdn.com/v4/letter/s/13edae/32.png) [@Sven\_Beauprez](https://discuss.elastic.co/u/Sven_Beauprez)\
**Post date:** [August 13, 2015, 2:55pm UTC](https://discuss.elastic.co/t/mapping-transform-only-for-creating-new-and-not-for-updating/27312/4 "2015-08-13T14:55:01Z")

</div>

@nik9000 I think I made a mistake in choosing my wording: I really do not want to change the \_source itself. I want to use the updated data, do some processing and 'add' that result to be indexed.

For example, an entry is made with a timestamp, an update is made with a new timestamp and I want to keep a duration between log statements. In other words in this case the difference between last and first timestamp. The \_source, which is updated, has the latest timestamp, which is exactly what I want. As an extra, I have a computed field that contains a duration (diff).

Does that suit the case of mapping transform or am I stretching things?

---

<div class="post-metadata">

**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\
**Post date:** [August 13, 2015, 3:23pm UTC](https://discuss.elastic.co/t/mapping-transform-only-for-creating-new-and-not-for-updating/27312/5 "2015-08-13T15:23:54Z")

</div>

> [@Sven\_Beauprez](#):
>
> Does that suit the case of mapping transform or am I stretching things?

If you want it to come back in the `_source` then its not going to work - if you just want to be able search for it then its fine. I think if you want a diff you probably want it in the `_source` though so I think I'd go with the update script. Transform is really for situations like "I want to copy my text field to my suggest field but only if my namespace field is 0".

---

<div class="post-metadata">

**Author:** ![Sven\_Beauprez](https://avatars.discourse-cdn.com/v4/letter/s/13edae/32.png) [@Sven\_Beauprez](https://discuss.elastic.co/u/Sven_Beauprez)\
**Post date:** [August 13, 2015, 3:28pm UTC](https://discuss.elastic.co/t/mapping-transform-only-for-creating-new-and-not-for-updating/27312/6 "2015-08-13T15:28:47Z")

</div>

> [@nik9000](#):
>
> If you want it to come back in the \_source then its not going to work - if you just want to be able search for it then its fine. I think if you want a diff you probably want it in the \_source though so I think I'd go with the update script. Transform is really for situations like "I want to copy my text field to my suggest field but only if my namespace field is 0".

Ok, got it.

Just to come back to the original question for completeness and future reference: it only works when creating new documents, not with updates of existing documents, am I right?

---

<div class="post-metadata">

**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\
**Post date:** [August 13, 2015, 5:33pm UTC](https://discuss.elastic.co/t/mapping-transform-only-for-creating-new-and-not-for-updating/27312/7 "2015-08-13T17:33:40Z")

</div>

> [@Sven\_Beauprez](#):
>
> Just to come back to the original question for completeness and future reference: it only works when creating new documents, not with updates of existing documents, am I right?

Its certainly supposed to work for updates.

---

<div class="post-metadata">

**Author:** ![Sven\_Beauprez](https://avatars.discourse-cdn.com/v4/letter/s/13edae/32.png) [@Sven\_Beauprez](https://discuss.elastic.co/u/Sven_Beauprez)\
**Post date:** [August 13, 2015, 7:10pm UTC](https://discuss.elastic.co/t/mapping-transform-only-for-creating-new-and-not-for-updating/27312/8 "2015-08-13T19:10:01Z")

</div>

I've made following simple test, both in ES 1.6 and 1.7 and the counter only updated when document was created:

> {  
> "mappings": {  
> "simple": {  
> "transform": {  
> "lang":"groovy",  
> "script":"ctx.\_source.counter = ctx.\_source.counter == null ? 1 : ctx.\_source.counter + 1"  
> },  
> "properties": {  
> "title": {  
> "type": "string"  
> },  
> "description": {  
> "type": "string"  
> },  
> "counter": {  
> "type": "long",  
> "store": "yes"  
> }  
> }  
> }  
> }  
> }

Then:

> PUT  
> { "title":"This is a document with text", "description":"null" }

When getting the document, the counter is correctly set to 1

When overwriting the document (PUT on same URL, not with \_update) the counter is still 1 while version is clearly increased (get with \_source\_transform):

> {  
> "\_index": "transform",  
> "\_type": "simple",  
> "\_id": "1",  
> "\_version": 3,  
> "found": true,  
> "\_source": {  
> "title": "This is a document with text",  
> "description": "null",  
> "counter": 1  
> }  
> }

Same when I update via POST with \_update URL

> POST .../\_update  
> { "doc" : { "title":"This is a document with text", "description":"null" }}

Result (get with \_source\_transform):

> {  
> "\_index": "transform",  
> "\_type": "simple",  
> "\_id": "1",  
> "\_version": 4,  
> "found": true,  
> "\_source": {  
> "title": "This is a document with text",  
> "description": "null",  
> "counter": 1  
> }  
> }

Am I making a stupid mistake here?

---

<div class="post-metadata">

**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\
**Post date:** [August 13, 2015, 7:59pm UTC](https://discuss.elastic.co/t/mapping-transform-only-for-creating-new-and-not-for-updating/27312/9 "2015-08-13T19:59:09Z")

</div>

OK. I had a think about this. The reason this happens is that changes from  
transform aren't saved to the source. They aren't supposed to be. They are  
for when you want to index stuff that doesn't match the source. Just like  
copy\_to.

---

<div class="post-metadata">

**Author:** ![Sven\_Beauprez](https://avatars.discourse-cdn.com/v4/letter/s/13edae/32.png) [@Sven\_Beauprez](https://discuss.elastic.co/u/Sven_Beauprez)\
**Post date:** [August 13, 2015, 8:03pm UTC](https://discuss.elastic.co/t/mapping-transform-only-for-creating-new-and-not-for-updating/27312/10 "2015-08-13T20:03:26Z")

</div>

IMO this is also true for updates. Do I need to open a github issue to discuss this?

Anyway, it is clear now what works and what doesn't. Thanks for your help!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 11:55pm UTC](https://discuss.elastic.co/t/mapping-transform-only-for-creating-new-and-not-for-updating/27312/11 "2017-07-05T23:55:55Z")

</div>