# Event.remove method not working inside aggregate section in code block **URL:** \ **Category:** Logstash\ **Created:** [June 16, 2023, 9:20am UTC](https://discuss.elastic.co/t/event-remove-method-not-working-inside-aggregate-section-in-code-block/336201 "2023-06-16T09:20:21Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![J\_S](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/j_s/32/122308_2.png) [@J\_S](https://discuss.elastic.co/u/J_S)\ **Post date:** [June 16, 2023, 9:20am UTC](https://discuss.elastic.co/t/event-remove-method-not-working-inside-aggregate-section-in-code-block/336201/1 "2023-06-16T09:20:21Z")
Hi All, I am newbie to ELK stack, I am trying to remove the field called "attributes" while aggregate the data inside code block. But it is not removing the already existing "attributes" in the corresponding "id" but only update the existing attributes. The actual expected result is I have to remove "attributes" if already available and create new field with same name "attributes", with selected results from SQL. Below is the code I tried so far: ```auto filter { if "my_index" in [tags] { aggregate { task_id => "%{employee.id}" code => " event.remove('attributes') map['id'] = event.get('employee.id') map['attributes'] ||= {} attributename = event.get('attributename') attributevalue = event.get('attributevalue') map['attributes'][attributename] = attributevalue event.cancel() " push_previous_map_as_event => true timeout => 3 } mutate { add_field => { "custom_index_name" => "my_index" } } } else { mutate { add_field => { "custom_index_name" => "%{type}" } } } } ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 16, 2023, 4:36pm UTC](https://discuss.elastic.co/t/event-remove-method-not-working-inside-aggregate-section-in-code-block/336201/2 "2023-06-16T16:36:02Z")
> [@J\_S](#): > > `event.remove('attributes')` This will remove the [attributes] field from the event that is being aggregated, but you then call event.cancel to delete that event, so it doesn't really matter. The event that is flushed from the map will have the [attributes] field your code aggregated into the map. ---
**Author:** ![J\_S](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/j_s/32/122308_2.png) [@J\_S](https://discuss.elastic.co/u/J_S)\ **Post date:** [June 16, 2023, 5:16pm UTC](https://discuss.elastic.co/t/event-remove-method-not-working-inside-aggregate-section-in-code-block/336201/3 "2023-06-16T17:16:52Z")
I tried it by removing event.cancel() but still I can't delete attributes existing already and create newly using map['attributes']. Could you please suggest how to change it in my code? Thanks. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 16, 2023, 5:24pm UTC](https://discuss.elastic.co/t/event-remove-method-not-working-inside-aggregate-section-in-code-block/336201/4 "2023-06-16T17:24:38Z")
> [@J\_S](#): > > I tried it by removing event.cancel() but still I can't delete attributes existing already and create newly using map['attributes']. I do not understand what you mean by this. The aggregate filter will create a new event containing whatever is in the map. That means it only has the fields that you have added to the map in your aggregate filter. ---
**Author:** ![J\_S](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/j_s/32/122308_2.png) [@J\_S](https://discuss.elastic.co/u/J_S)\ **Post date:** [June 16, 2023, 5:53pm UTC](https://discuss.elastic.co/t/event-remove-method-not-working-inside-aggregate-section-in-code-block/336201/5 "2023-06-16T17:53:15Z")
Yeah Thanks! I got it, this aggregated data will be saved and to prevent other event to access it. If so, how to update the map['attributes'] with newly fetched values. Suppose I've below already inside attributes: ```auto attributes: { A: 1, B: 2, C:3, D:4} attributes comes from SQL: {A: 2, B:5 C: 1} ``` In above code A, B, C is updating properly. But D:4 is still there it's not able to delete. So I am trying to remove attributes entirely and create a new attributes field with values comes from SQL. Can you please help me on this? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 16, 2023, 8:14pm UTC](https://discuss.elastic.co/t/event-remove-method-not-working-inside-aggregate-section-in-code-block/336201/6 "2023-06-16T20:14:52Z")
D:4 will only be there is there is an event for that employee.id that contains ``` "attributename" => "D" "attributevalue" => "4" ``` ---
**Author:** ![J\_S](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/j_s/32/122308_2.png) [@J\_S](https://discuss.elastic.co/u/J_S)\ **Post date:** [June 17, 2023, 2:33am UTC](https://discuss.elastic.co/t/event-remove-method-not-working-inside-aggregate-section-in-code-block/336201/7 "2023-06-17T02:33:38Z")
Thanks for clarification, in this case how we can replace existing attributes completely by new attributes from SQL? I mean how to replace above attributes A,B,C,D (existing) with A,B,C (from SQL) ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 17, 2023, 3:19am UTC](https://discuss.elastic.co/t/event-remove-method-not-working-inside-aggregate-section-in-code-block/336201/8 "2023-06-17T03:19:01Z")
I am still confused, but just had another idea. Are you saying that you have an existing document in elasticsearch where the [attributes] field is "{ A: 1, B: 2, C:3, D:4}", and if you fetch a new row from the jdbc input that has "{A: 2, B:5 C: 1}" you want to replace the [attributes] field on the document with whatever comes out of logstash? If so, aggregate is not the way to go. Do you need to overwrite the entire document, or just replace the [attributes] field? You will probably need to ask a new question to get an answer. If you answer the questions above I can help you tomorrow with what to ask in the new question. ---
**Author:** ![J\_S](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/j_s/32/122308_2.png) [@J\_S](https://discuss.elastic.co/u/J_S)\ **Post date:** [June 17, 2023, 4:55am UTC](https://discuss.elastic.co/t/event-remove-method-not-working-inside-aggregate-section-in-code-block/336201/9 "2023-06-17T04:55:35Z")
> [@Badger](#): > > Are you saying that you have an existing document in elasticsearch where the [attributes] field is "{ A: 1, B: 2, C:3, D:4}", and if you fetch a new row from the jdbc input that has "{A: 2, B:5 C: 1}" you want to replace the [attributes] field on the document with whatever comes out of logstash? Yes, I need to vanish existing data completely inside attributes object. After that, I need to insert into attributes with the values comes from SQL. Hope this can clear now @Badger . Could you please alter above logstash.conf for this case? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 17, 2023, 4:20pm UTC](https://discuss.elastic.co/t/event-remove-method-not-working-inside-aggregate-section-in-code-block/336201/10 "2023-06-17T16:20:34Z")
Still need a bit more detail. You can overwrite the document in elasticsearch by use `document_id => "%{employee.id}"`. However, if you have additional fields and logstash is only processing [attributes] overwriting the document will lose all the other fields. That's why I asked: Do you need to overwrite the entire document, or just replace the [attributes] field? ---
**Author:** ![J\_S](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/j_s/32/122308_2.png) [@J\_S](https://discuss.elastic.co/u/J_S)\ **Post date:** [June 18, 2023, 12:49am UTC](https://discuss.elastic.co/t/event-remove-method-not-working-inside-aggregate-section-in-code-block/336201/11 "2023-06-18T00:49:56Z")
HI @Badger , Yes I need to overwrite the document using ID. But those fields are overwrite properly using above code. If new attributes are available, I need to replace it completely using ID. I am worry about attributes field alone not overwriting with new data. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 18, 2023, 1:34am UTC](https://discuss.elastic.co/t/event-remove-method-not-working-inside-aggregate-section-in-code-block/336201/12 "2023-06-18T01:34:05Z")
Use `output { stdout }` and show us the event then show us the same document in elasticsearch. ---
**Author:** ![J\_S](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/j_s/32/122308_2.png) [@J\_S](https://discuss.elastic.co/u/J_S)\ **Post date:** [June 18, 2023, 1:54am UTC](https://discuss.elastic.co/t/event-remove-method-not-working-inside-aggregate-section-in-code-block/336201/13 "2023-06-18T01:54:41Z")
Hi @Badger , This is the format of the event in stdout. ```auto { "custom_index_name" => "my_index", "@version" => "1", "id" => 375283, "name" => "Test Data", "@timestamp" => 2023-04-24T09:56:47.000Z, "attributes" => { "Name" => "Jack", "Age" => "27", "City" => "Chennai" }, "tags" => [ [0] "_aggregatefinalflush" ] } ``` In Elastic search the document is like below: ```auto { "took" : 4, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 1, "relation" : "eq" }, "max_score" : 1.0, "hits" : [ { "_index" : "my_index", "_id" : "375283", "_score" : 1.0, "_source" : { "tags" : [ "_aggregatefinalflush" ], "@timestamp" : "2023-04-24T09:56:47.000Z", "attributes" : { "Name" => "Jack", "Age" => "27", "City" => "Chennai" }, "@version" : "1", "name" : "Test Data", "id" : 375283, "custom_index_name" : "my_index" } } ] } } ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 18, 2023, 2:20am UTC](https://discuss.elastic.co/t/event-remove-method-not-working-inside-aggregate-section-in-code-block/336201/14 "2023-06-18T02:20:48Z")
OK, so the final flush event has three attributes, and elasticsearch has same three attributes. I do not see an issue. ---
**Author:** ![J\_S](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/j_s/32/122308_2.png) [@J\_S](https://discuss.elastic.co/u/J_S)\ **Post date:** [June 18, 2023, 2:27am UTC](https://discuss.elastic.co/t/event-remove-method-not-working-inside-aggregate-section-in-code-block/336201/15 "2023-06-18T02:27:12Z")
Yes @Badger now it looks same count. But if I re run and SQL returns more than 3 items which are new ones, the existing 3 items are not getting deleted. But I need those 3 attributes needs to be delete. That's the issue I'm facing. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 18, 2023, 2:39am UTC](https://discuss.elastic.co/t/event-remove-method-not-working-inside-aggregate-section-in-code-block/336201/16 "2023-06-18T02:39:16Z")
What does the elasticsearch output look like in your logstash configuration? ---
**Author:** ![J\_S](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/j_s/32/122308_2.png) [@J\_S](https://discuss.elastic.co/u/J_S)\ **Post date:** [June 18, 2023, 2:47am UTC](https://discuss.elastic.co/t/event-remove-method-not-working-inside-aggregate-section-in-code-block/336201/17 "2023-06-18T02:47:38Z")
This is my output filter part. ```auto output { elasticsearch { hosts => ["http://localhost:9200"] index => "my_index" document_id => "%{id}" document_type => "%{type}" action => "update" doc_as_upsert => true } } ``` Here I am trying to update Index, the other fields mentioned above are update/override properly. But attributes field alone is not update as per result in SQL. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 18, 2023, 2:51am UTC](https://discuss.elastic.co/t/event-remove-method-not-working-inside-aggregate-section-in-code-block/336201/18 "2023-06-18T02:51:55Z")
> [@J\_S](#): > > `doc_as_upsert => true` I think you have misunderstood what that does. See [this](https://discuss.elastic.co/t/prevent-upsert-to-merge-internal-objects-in-logstash-configuration/148061) thread. I suggest you ask a new question, saying: I am repeatedly fetching rows from a database. I insert them into elasticsearch using the unique key as the document\_id. For any fields not on the current document I want to add any missing columns to the exiting documents. For fields that _do_ exist on the current document (including hashes) I want to overwrite them. How can I do this? ---
**Author:** ![J\_S](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/j_s/32/122308_2.png) [@J\_S](https://discuss.elastic.co/u/J_S)\ **Post date:** [June 18, 2023, 3:11am UTC](https://discuss.elastic.co/t/event-remove-method-not-working-inside-aggregate-section-in-code-block/336201/19 "2023-06-18T03:11:02Z")
I've created the new topic [here](https://discuss.elastic.co/t/updating-index-is-not-working-for-existing-data-inside-json-object/336291) with same details as you've suggested. The link you've shared above is also exact similar case I am trying to achieve it. Kindly help me on this @Badger . Thanks ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 18, 2023, 3:31am UTC](https://discuss.elastic.co/t/event-remove-method-not-working-inside-aggregate-section-in-code-block/336201/20 "2023-06-18T03:31:12Z")
I cannot help at this point (I do not even run elasticsearch), but you have now posted a much clearer question that I suspect one of the other regulars can respond to. Good luck! [Next page](https://discuss.elastic.co/t/event-remove-method-not-working-inside-aggregate-section-in-code-block/336201.md?page=2)