# Updating index is not working for existing data inside json object

**URL:** <https://discuss.elastic.co/t/updating-index-is-not-working-for-existing-data-inside-json-object/336291>\
**Category:** Logstash\
**Created:** [June 18, 2023, 3:06am UTC](https://discuss.elastic.co/t/updating-index-is-not-working-for-existing-data-inside-json-object/336291 "2023-06-18T03:06:17Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![J\_S](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/j_s/32/122308_2.png) [@J\_S](https://discuss.elastic.co/u/J_S)\
**Post date:** [June 18, 2023, 3:06am UTC](https://discuss.elastic.co/t/updating-index-is-not-working-for-existing-data-inside-json-object/336291/1 "2023-06-18T03:06:17Z")

</div>

I am repeatedly fetching rows from a database. I insert them into elasticsearch using the unique key as the `document_id`. For any fields not on the current document I want to add any missing columns to the exiting documents. For fields that _do_ exist on the current document (including hashes) I want to overwrite them. How can I do this?

Here is my filter and output part:

```auto
filter {

		
       if "my_index" in [tags] {
	   
	   
	   
        aggregate {
            task_id => "%{componentinstance.id}"
            code => "
                map['id'] = event.get('componentinstance.id')
                map['name'] = 'Test Data'
                map['@timestamp'] = event.get('@timestamp')
                map['attributes'] ||= {}
				map['attributes'][event.get('attributename')] = event.get('attributevalue')
                event.cancel()
            "
            push_previous_map_as_event => true
            timeout => 3
        }

        mutate {
            add_field => { "custom_index_name" => "my_index" }
        }
		
    } else {
        mutate {
            add_field => { "custom_index_name" => "%{type}" }
        }
    }
}
output {
     elasticsearch {
      hosts => ["http://localhost:9200"]
      index => "my_index"
      document_id => "%{id}"
      document_type => "%{type}"
      action => "update"
      doc_as_upsert => true
    }
}

Output from stdout:

{
                 "tags" => [
        [0] "_aggregatefinalflush"
    ],
           "@timestamp" => 2023-04-24T09:56:47.000Z,
           "attributes" => {
                 "Name" => "Jack",
                "Age" => "27",
        "City" => "Chennai"
    },
             "@version" => "1",
                 "name" => "Test Data",
                   "id" => 375283,
    "custom_index_name" => "my_index"
}

```

If new attributes are getting added from SQL, the existing attributes remains same and it won't get removed.

Old Data:

```auto
 "attributes" => {
              "Name" => "Jack",
             "Age" => "27",
             "City" => "Chennai"
 }

```

New Data:

```auto
       "attributes" => {
                 "Name" => "Sunny",
                "Age" => "24"
    }

```

Current output:

```auto
       "attributes" => {
                 "Name" => "Sunny",
                "Age" => "24",
                "City" => "Chennai" / ***It is not removed now, this was the issue*** /
    }

```

Expected Output:

```auto
       "attributes" => {
                 "Name" => "Sunny",
                "Age" => "24"
    }

```

---

<div class="post-metadata">

**Author:** ![J\_S](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/j_s/32/122308_2.png) [@J\_S](https://discuss.elastic.co/u/J_S)\
**Post date:** [June 18, 2023, 7:15am UTC](https://discuss.elastic.co/t/updating-index-is-not-working-for-existing-data-inside-json-object/336291/2 "2023-06-18T07:15:53Z")

</div>

Hi @Badger ,

Can you please help me on this issue I've posted as a new topic as discussed.😊😊

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 16, 2023, 7:16am UTC](https://discuss.elastic.co/t/updating-index-is-not-working-for-existing-data-inside-json-object/336291/3 "2023-07-16T07:16:16Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.