# Weird behaviour on document update to remove value from a field

**URL:** <https://discuss.elastic.co/t/weird-behaviour-on-document-update-to-remove-value-from-a-field/363427>\
**Category:** Elasticsearch\
**Tags:** language-clients\
**Created:** [July 19, 2024, 11:32am UTC](https://discuss.elastic.co/t/weird-behaviour-on-document-update-to-remove-value-from-a-field/363427 "2024-07-19T11:32:36Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![Vijeya\_Nidhi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vijeya_nidhi/32/123510_2.png) [@Vijeya\_Nidhi](https://discuss.elastic.co/u/Vijeya_Nidhi)\
**Post date:** [July 19, 2024, 11:32am UTC](https://discuss.elastic.co/t/weird-behaviour-on-document-update-to-remove-value-from-a-field/363427/1 "2024-07-19T11:32:36Z")

</div>

Elasticsearch version 8.13.3

Assume there is a document with id `test` in index `testindex` with following fields.

```auto
{
  "field1": "any1",
  "field2": "any2"
}

```

if i execute a update request like  
POST: `<hostname>/testindex/_update/test?refresh=false`  
Body:

```auto
{
	"doc": {
		"field2": null
	},
	"doc_as_upsert": true
}

```

The document gets updated to

```auto
{
  "field1": "any1",
  "field2": null
}

```

But the same if we execute via the client, value in the field `field2` does not get updated to null.

```auto
Map<String,Object> testbody = new HashMap<>();
        testbody.put("field2",null);

UpdateRequest<Map<String,Object>,Map<String,Object>> testRequest = new UpdateRequest.Builder<Map<String,Object>,Map<String,Object>>()
                .id("test")
                .index("testindex")
                .doc(testbody)
                .docAsUpsert(true)
                .build();
client.update(testRequest,HashMap.class);

```

Any idea why the client behaves like this?

---

<div class="post-metadata">

**Author:** ![ltrotta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ltrotta/32/124003_2.png) [@ltrotta](https://discuss.elastic.co/u/ltrotta)\
**Post date:** [July 19, 2024, 12:57pm UTC](https://discuss.elastic.co/t/weird-behaviour-on-document-update-to-remove-value-from-a-field/363427/2 "2024-07-19T12:57:51Z")

</div>

Hello!  
This is weird, I have tried this with server version 8.14.0 and java client versions 8.13.3 and the latest 8.14.3, and it works as expected, `field2` gets updated to `null`. Could you give me more information on which client + server versions are you using?

---

<div class="post-metadata">

**Author:** ![Vijeya\_Nidhi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vijeya_nidhi/32/123510_2.png) [@Vijeya\_Nidhi](https://discuss.elastic.co/u/Vijeya_Nidhi)\
**Post date:** [July 19, 2024, 1:28pm UTC](https://discuss.elastic.co/t/weird-behaviour-on-document-update-to-remove-value-from-a-field/363427/3 "2024-07-19T13:28:19Z")

</div>

The server version is 8.13.4 ([https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.13.4-darwin-x86\_64.tar.gz](https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.13.4-darwin-x86_64.tar.gz)) and the client version is 8.13.3 ([https://mvnrepository.com/artifact/co.elastic.clients/elasticsearch-java/8.13.3](https://mvnrepository.com/artifact/co.elastic.clients/elasticsearch-java/8.13.3)).

---

<div class="post-metadata">

**Author:** ![Vijeya\_Nidhi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vijeya_nidhi/32/123510_2.png) [@Vijeya\_Nidhi](https://discuss.elastic.co/u/Vijeya_Nidhi)\
**Post date:** [July 19, 2024, 1:48pm UTC](https://discuss.elastic.co/t/weird-behaviour-on-document-update-to-remove-value-from-a-field/363427/4 "2024-07-19T13:48:57Z")

</div>

Also just now tried the code against the newer es version both server and java elastic client as 8.14.3

The issue still existed.

Also just to be safe I logged out the request (`logger.info("{}"),request`) and this was the update request which got generated and executed.

`UpdateRequest: POST /testindex/_update/test?refresh=false {"doc":"{field1=any1, field2=null}","doc_as_upsert":true}`

But even after this was executed and the index refreshed, the `field2` was still present with the same value (`any2`) in the document.

---

<div class="post-metadata">

**Author:** ![RabBit\_BR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rabbit_br/32/82261_2.png) [@RabBit\_BR](https://discuss.elastic.co/u/RabBit_BR)\
**Post date:** [July 19, 2024, 2:18pm UTC](https://discuss.elastic.co/t/weird-behaviour-on-document-update-to-remove-value-from-a-field/363427/5 "2024-07-19T14:18:32Z")

</div>

Strange. The way below works but it would be good to understand why your way failed, maybe open an issue.

`.doc(JsonData.fromJson(new ObjectMapper().writeValueAsString(testbody)));`

---

<div class="post-metadata">

**Author:** ![ltrotta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ltrotta/32/124003_2.png) [@ltrotta](https://discuss.elastic.co/u/ltrotta)\
**Post date:** [July 19, 2024, 2:22pm UTC](https://discuss.elastic.co/t/weird-behaviour-on-document-update-to-remove-value-from-a-field/363427/6 "2024-07-19T14:22:36Z")

</div>

I wrote a quick reproducer, could you execute this and let me know the output?

```java
Map<String,Object> testbody = new HashMap<>();
testbody.put("field1","any1");
testbody.put("field2","any2");

esClient.index(ix -> ix.index("test-upsert-null").id("test").document(testbody).refresh(Refresh.WaitFor));

SearchResponse<HashMap> res = esClient.search(s -> s.index("test-upsert-null"),HashMap.class);
System.out.println("current field2 value: " + res.hits().hits().get(0).source().get("field2"));

testbody.put("field2",null);

esClient.update(ur -> ur.id("test").doc(testbody).index("test-upsert-null").docAsUpsert(true).refresh(Refresh.WaitFor),HashMap.class);
res = esClient.search(s -> s.index("test-upsert-null"),HashMap.class);
System.out.println("final field2 value: " + res.hits().hits().get(0).source().get("field2"));

```

---

<div class="post-metadata">

**Author:** ![Vijeya\_Nidhi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vijeya_nidhi/32/123510_2.png) [@Vijeya\_Nidhi](https://discuss.elastic.co/u/Vijeya_Nidhi)\
**Post date:** [July 19, 2024, 2:33pm UTC](https://discuss.elastic.co/t/weird-behaviour-on-document-update-to-remove-value-from-a-field/363427/7 "2024-07-19T14:33:22Z")

</div>

Hey,

this was the output

```auto
current field2 value: any2
final field2 value: any2

```

---

<div class="post-metadata">

**Author:** ![Vijeya\_Nidhi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vijeya_nidhi/32/123510_2.png) [@Vijeya\_Nidhi](https://discuss.elastic.co/u/Vijeya_Nidhi)\
**Post date:** [July 19, 2024, 2:35pm UTC](https://discuss.elastic.co/t/weird-behaviour-on-document-update-to-remove-value-from-a-field/363427/8 "2024-07-19T14:35:09Z")

</div>

but interestingly, the fix given by @RabBit_BR worked.

```auto
Map<String,Object> testbody = new HashMap<>();
        testbody.put("field1","any1");
        testbody.put("field2","any2");

        client.index(ix -> ix.index("test-upsert-null").id("test").document(testbody).refresh(Refresh.WaitFor));

        SearchResponse<HashMap> res = client.search(s -> s.index("test-upsert-null"),HashMap.class);
        System.out.println("current field2 value: " + res.hits().hits().get(0).source().get("field2"));

        testbody.put("field2",null);

        client.update(ur -> {
            try {
                return ur.id("test").doc(JsonData.fromJson(new ObjectMapper().writeValueAsString(testbody))).index("test-upsert-null").docAsUpsert(true).refresh(Refresh.WaitFor);
            } catch (JsonProcessingException e) {
                throw new RuntimeException(e);
            }
        },HashMap.class);
        res = client.search(s -> s.index("test-upsert-null"),HashMap.class);
        System.out.println("final field2 value: " + res.hits().hits().get(0).source().get("field2"));

```

output

```auto
current field2 value: any2
final field2 value: null

```

---

<div class="post-metadata">

**Author:** ![ltrotta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ltrotta/32/124003_2.png) [@ltrotta](https://discuss.elastic.co/u/ltrotta)\
**Post date:** [July 19, 2024, 2:36pm UTC](https://discuss.elastic.co/t/weird-behaviour-on-document-update-to-remove-value-from-a-field/363427/9 "2024-07-19T14:36:33Z")

</div>

Do you perhaps have any custom configuration on the json mapper used to create the elasticsearch client?

---

<div class="post-metadata">

**Author:** ![Vijeya\_Nidhi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vijeya_nidhi/32/123510_2.png) [@Vijeya\_Nidhi](https://discuss.elastic.co/u/Vijeya_Nidhi)\
**Post date:** [July 19, 2024, 2:39pm UTC](https://discuss.elastic.co/t/weird-behaviour-on-document-update-to-remove-value-from-a-field/363427/10 "2024-07-19T14:39:29Z")

</div>

Unless I am making a mistake in the below code.  
I would say I am not using any custom configuration.

```auto
builder = RestClient
                .builder(new HttpHost(this.targetUrl.getHost(), this.targetUrl.getPort(), this.targetUrl.getScheme()));

        RestClient lowLevelRestClient = builder.build();
        transport = new RestClientTransport(
                lowLevelRestClient,
                new JacksonJsonpMapper()
        );
        clientES = new ElasticsearchClient(transport);

```

---

<div class="post-metadata">

**Author:** ![RabBit\_BR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rabbit_br/32/82261_2.png) [@RabBit\_BR](https://discuss.elastic.co/u/RabBit_BR)\
**Post date:** [July 19, 2024, 2:40pm UTC](https://discuss.elastic.co/t/weird-behaviour-on-document-update-to-remove-value-from-a-field/363427/11 "2024-07-19T14:40:52Z")

</div>

Great point @ltrotta . @Vijeya_Nidhi , change as below and test your original code.

```
    JacksonJsonpMapper mapper = new JacksonJsonpMapper();
    mapper.objectMapper().setSerializationInclusion(JsonInclude.Include.ALWAYS);
    ElasticsearchTransport transport = new RestClientTransport(
            restClient, mapper);

```

---

<div class="post-metadata">

**Author:** ![Vijeya\_Nidhi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vijeya_nidhi/32/123510_2.png) [@Vijeya\_Nidhi](https://discuss.elastic.co/u/Vijeya_Nidhi)\
**Post date:** [July 19, 2024, 2:45pm UTC](https://discuss.elastic.co/t/weird-behaviour-on-document-update-to-remove-value-from-a-field/363427/12 "2024-07-19T14:45:44Z")

</div>

yup the mapper change suggested worked.

output

```auto
current field2 value: any2
final field2 value: null

```

Thanks for the help.

---

<div class="post-metadata">

**Author:** ![ltrotta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ltrotta/32/124003_2.png) [@ltrotta](https://discuss.elastic.co/u/ltrotta)\
**Post date:** [July 19, 2024, 2:48pm UTC](https://discuss.elastic.co/t/weird-behaviour-on-document-update-to-remove-value-from-a-field/363427/13 "2024-07-19T14:48:20Z")

</div>

Thank you both for the help, yes the issue was with the default JacksonJsonpMapper, by initializing the JacksonJsonpMapper without passing any ObjectMapper, the default used constructor will build:

```java
    public JacksonJsonpMapper() {
        this((new ObjectMapper()).configure(SerializationFeature.INDENT_OUTPUT, false).setSerializationInclusion(Include.NON_NULL));
    }

```

which ignores `null` value in the serialization. I assume you got the code from the official documentation, we'll update it with the recommended way to initialize the mapper.

---

<div class="post-metadata">

**Author:** ![Vijeya\_Nidhi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vijeya_nidhi/32/123510_2.png) [@Vijeya\_Nidhi](https://discuss.elastic.co/u/Vijeya_Nidhi)\
**Post date:** [July 19, 2024, 2:52pm UTC](https://discuss.elastic.co/t/weird-behaviour-on-document-update-to-remove-value-from-a-field/363427/14 "2024-07-19T14:52:53Z")

</div>

yup, we are in the process of migrating our code base from high level rest client to the recommended java api client, so for the initialisation purpose we just used the code snippet in the documentation to initialise the client.

but it does makes sense looking back, why until the request is generated, it was having null in the body, but on execution of the request it failed to remove the value from the field.

Any thanks to both of you for the quick help and pointing out where the problem was.