# How to remove fields in logstash output

**URL:** <https://discuss.elastic.co/t/how-to-remove-fields-in-logstash-output/173469>\
**Category:** Logstash\
**Created:** [March 22, 2019, 9:54am UTC](https://discuss.elastic.co/t/how-to-remove-fields-in-logstash-output/173469 "2019-03-22T09:54:05Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![evahsiao](https://avatars.discourse-cdn.com/v4/letter/e/d26b3c/32.png) [@evahsiao](https://discuss.elastic.co/u/evahsiao)\
**Post date:** [March 22, 2019, 9:54am UTC](https://discuss.elastic.co/t/how-to-remove-fields-in-logstash-output/173469/1 "2019-03-22T09:54:06Z")

</div>

I want to pass my json data from kafka to elasticseaarch. Judged by the operation param to different action of elasticsearch.  
my json data is like this:  
{  
"operation": "create",  
"fields": {  
"sign": "im\_messages",  
"from\_id": 12797,  
"to\_id": 12796,  
"content": "hello",  
"accept": 0,  
"is\_revoke": 0,  
"created\_at": "2019-03-21 13:53:16"  
}  
}  
now i have successfully transfer the data into es by different operation,such as create/update/delete/index. The problem is, I cant remove the operation param from elasticsearch, because if i remove operation in the filter, then i will cant use it for the output elasticsearch action. How can i do it?  
my logstash filter and output config is like this:  
filter {  
mutate {  
add\_field =\> {"@fields" =\> "%{fields}"}  
}  
json {  
source =\> "@fields"  
remove\_field =\> ["@fields","@version","@timestamp","fields"]  
}  
}

output {  
if [operation] == "create" {  
elasticsearch {  
hosts =\> ["xx.xx.xx.xx:9200"]  
index =\> "im"  
timeout =\> 300  
#user =\> "elastic"  
#password =\> "changeme"  
}  
.....  
and now the update and create operation result in elasticsearch is like this:  
-{  
"\_id": "oyl5pGkBReZGJ\_hya8ck",  
"\_index": "im",  
"\_score": 1,  
-"\_source": {  
"accept": 0,  
"content": "ssssssssssssss",  
"created\_at": "2019-03-22 16:14:45",  
"from\_id": 12796,  
"id": "oyl5pGkBReZGJ\_hya8ck",  
"is\_revoke": 0,  
"operation": "update",  
"sign": "im\_messages",  
"to\_id": 12797  
},  
"\_type": "doc"  
},  
-{  
"\_id": "An-BpGkBUAnQ0IgRfic8",  
"\_index": "im",  
"\_score": 1,  
-"\_source": {  
"accept": 0,  
"content": "???",  
"created\_at": "2019-03-22 10:21:56",  
"from\_id": 12796,  
"is\_revoke": 0,  
"operation": "create",  
"sign": "im\_messages",  
"to\_id": 12797  
},  
"\_type": "doc"  
}  
The operation and id params is what i want to remove in the source field.  
what can i do?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 22, 2019, 1:10pm UTC](https://discuss.elastic.co/t/how-to-remove-fields-in-logstash-output/173469/2 "2019-03-22T13:10:04Z")

</div>

If you want to sprintf reference a field in the output but do not want it as part of the event then move it to [@metadata].

```
 mutate { rename => { "[operation]" => "[@metadata][operation]" } }
```

---

<div class="post-metadata">

**Author:** ![evahsiao](https://avatars.discourse-cdn.com/v4/letter/e/d26b3c/32.png) [@evahsiao](https://discuss.elastic.co/u/evahsiao)\
**Post date:** [March 25, 2019, 7:01am UTC](https://discuss.elastic.co/t/how-to-remove-fields-in-logstash-output/173469/5 "2019-03-25T07:01:25Z")

</div>

yeah, it works, but new problem occurs.  
I add metadata in the filter:  
filter {  
mutate {  
add\_field =\> {"@fields" =\> "%{fields}"}  
rename =\> { "[operation]" =\> "[@metadata][operation]" }  
rename =\> { "[id]" =\> "[@metadata][id]" }  
}  
json {  
source =\> "@fields"  
remove\_field =\> ["@fields","@version","@timestamp","fields"]  
}  
}  
and i use the id metadata to update a doc when the operation is set to update:  
if [@metadata][operation] == "update" {  
elasticsearch {  
hosts =\> ["xx.xx.xx.xx:9200"]  
index =\> "im"  
action =\> "update"  
doc\_as\_upsert =\> true  
timeout =\> 300  
document\_id =\> "%{[@metadata][id]}"  
#user =\> "elastic"  
#password =\> "changeme"  
}  
}

the result is create a new doc other than update the exists one:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/9/293a100904bfb6b16791a0194430e9d28db99ea7.png)

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 25, 2019, 12:53pm UTC](https://discuss.elastic.co/t/how-to-remove-fields-in-logstash-output/173469/6 "2019-03-25T12:53:19Z")

</div>

That suggests that the [id] field does not exist at the point where you do the mutate+rename. Is it created by the json filter? If so, move the rename+mutate of [id] _after_ the json filter.

Also, please do not post pictures of text. Just post the text.

---

<div class="post-metadata">

**Author:** ![evahsiao](https://avatars.discourse-cdn.com/v4/letter/e/d26b3c/32.png) [@evahsiao](https://discuss.elastic.co/u/evahsiao)\
**Post date:** [March 25, 2019, 12:58pm UTC](https://discuss.elastic.co/t/how-to-remove-fields-in-logstash-output/173469/7 "2019-03-25T12:58:59Z")

</div>

No, the [id] field is not created by json filter ,it is tranfer from the massage data, and i want the [id] for elasticsearch document\_id which depends which doc to update, but not added to the source field.  
my json data for update is like this:  
{"operation":"update","fields":{"id":"sCmcs2kBReZGJ\_hybsd4","content":"xxxxxxxxxxxxx","updated\_at":"2019-03-25 14:14:45"}}  
and i want to update this doc:  
{

- "\_id": "sCmcs2kBReZGJ\_hybsd4",
- "\_index": "im",
- "\_score": 1,
- 
  - 

"\_source": {

- "accept": 0,
- "content": "bbbbbbbbb",
- "created\_at": "2019-03-25 10:12:56",
- "from\_id": 12796,
- "is\_revoke": 0,
- "sign": "im\_messages",
- "to\_id": 12797},
- "\_type": "doc"

},

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 25, 2019, 2:19pm UTC](https://discuss.elastic.co/t/how-to-remove-fields-in-logstash-output/173469/8 "2019-03-25T14:19:51Z")

</div>

> [@evahsiao](#):
>
> No, the [id] field is not created by json filter ,it is tranfer from the massage data

So when you say "No", you mean "Yes". The [id] field does not exist until the json filter is executed. Move the rename+mutate of [id] _after_ the json filter

---

<div class="post-metadata">

**Author:** ![evahsiao](https://avatars.discourse-cdn.com/v4/letter/e/d26b3c/32.png) [@evahsiao](https://discuss.elastic.co/u/evahsiao)\
**Post date:** [March 26, 2019, 1:17am UTC](https://discuss.elastic.co/t/how-to-remove-fields-in-logstash-output/173469/9 "2019-03-26T01:17:06Z")

</div>

Yeah, I put the codec to json in the input settings, and remove the rename+mutate of [id], this can update the exact doc of elasticsearch, but the document source has a [id] field which i dont want to set in the source.  
my logstash config file is setting like this:

input {  
kafka {  
bootstrap\_servers =\> "xx.xx.xx.xx:9092"  
#client\_id =\> "logstash1" # The purpose of this is to be able to track the source of requests beyond just ip/port by allowing a logical application name to be included.  
group\_id =\> "im\_group"  
auto\_offset\_reset =\> "latest" #automatically reset the offset to the latest offset  
consumer\_threads =\> 3  
topics =\> ["im"]  
#type =\> "message"  
codec =\> json {  
charset =\> "UTF-8"  
}  
}  
}

filter {  
mutate {  
add\_field =\> {"@fields" =\> "%{fields}"}  
rename =\> { "[operation]" =\> "[@metadata][operation]" }  
#rename =\> { "[id]" =\> "[@metadata][id]" }  
}  
json {  
source =\> "@fields"  
remove\_field =\> ["@fields","@version","@timestamp","fields"]  
}  
}

output {  
if [@metadata][operation] == "create" {  
elasticsearch {  
hosts =\> ["xx.xx.xx.xx:9200"]  
index =\> "im"  
timeout =\> 300  
#user =\> "elastic"  
#password =\> "changeme"  
}  
} else if [@metadata][operation] == "update" {  
elasticsearch {  
hosts =\> ["xx.xx.xx.xx:9200"]  
index =\> "im"  
action =\> "update"  
doc\_as\_upsert =\> true  
timeout =\> 300  
document\_id =\> "%{[id]}"  
#document\_id =\> "%{[@metadata][id]}"  
#user =\> "elastic"  
#password =\> "changeme"  
}  
the result of update is like this:  
{

- "\_id": "Bn-Jt2kBUAnQ0IgRcSc2",
- "\_index": "im",
- "\_score": 1,
- 
  - 

"\_source": {

- "accept": 0,
- "content": "xxxxxxxxxxxxx",
- "created\_at": "2019-03-25 10:12:56",
- "from\_id": 12796,
- "id": "Bn-Jt2kBUAnQ0IgRcSc2",
- "is\_revoke": 0,
- "sign": "im\_messages",
- "to\_id": 12797,
- "updated\_at": "2019-03-25 14:14:45"},
- "\_type": "doc"

}  
the [id] field in both the doc and the inside source of doc, i dont want it to add to the source

---

<div class="post-metadata">

**Author:** ![evahsiao](https://avatars.discourse-cdn.com/v4/letter/e/d26b3c/32.png) [@evahsiao](https://discuss.elastic.co/u/evahsiao)\
**Post date:** [April 10, 2019, 9:27am UTC](https://discuss.elastic.co/t/how-to-remove-fields-in-logstash-output/173469/10 "2019-04-10T09:27:03Z")

</div>

Finally, I give up this issue, and i find these extra fields have no influence in my normal function works. Thanks 😃

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 8, 2019, 9:27am UTC](https://discuss.elastic.co/t/how-to-remove-fields-in-logstash-output/173469/11 "2019-05-08T09:27:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.