# How to remove a filed

**URL:** <https://discuss.elastic.co/t/how-to-remove-a-filed/105173>\
**Category:** Logstash\
**Created:** [October 25, 2017, 6:57am UTC](https://discuss.elastic.co/t/how-to-remove-a-filed/105173 "2017-10-25T06:57:48Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![raghu1](https://avatars.discourse-cdn.com/v4/letter/r/ec9cab/32.png) [@raghu1](https://discuss.elastic.co/u/raghu1)\
**Post date:** [October 25, 2017, 6:57am UTC](https://discuss.elastic.co/t/how-to-remove-a-filed/105173/1 "2017-10-25T06:57:48Z")

</div>

Hi Team,

Please can you suggest how to remove field "payload" from below pattern:

I am trying using remove\_field =\> ["payload"]. Please suggest.

{match =\> { "message" =\> "[%{GREEDYDATA:BEGIN}]|%{GREEDYDATA:service\_id}|%{GREEDYDATA:log\_level}|%{GREEDYDATA:log\_type}|%{GREEDYDATA:log\_code}|%{GREEDYDATA:log\_message}|%{GREEDYDATA:log\_guid}|%{GREEDYDATA:org\_code}|%{GREEDYDATA:process\_name}|%{GREEDYDATA:si\_id}|%{GREEDYDATA:operation\_name}|%{GREEDYDATA:service\_requestor}|%{GREEDYDATA:service\_provider}|%{GREEDYDATA:machine\_name}|%{GREEDYDATA:engine\_name}|%{GREEDYDATA:payload}|%{GREEDYDATA:pre\_step}|%{GREEDYDATA:post\_step}|%{GREEDYDATA:log\_timestamp}|%{GREEDYDATA:si\_id}|%{GREEDYDATA:EOD}  
remove\_field =\> ["payload"]

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 25, 2017, 7:53am UTC](https://discuss.elastic.co/t/how-to-remove-a-filed/105173/2 "2017-10-25T07:53:32Z")

</div>

If you don't want to catch the `payload` field just replace `%{GREEDYDATA:payload}` with `%{GREEDYDATA}`.

But that grok expression is awful. That amount of GREEDYDATA patterns is extremely inefficient and could give the wrong results. Just use a csv filter.

---

<div class="post-metadata">

**Author:** ![raghu1](https://avatars.discourse-cdn.com/v4/letter/r/ec9cab/32.png) [@raghu1](https://discuss.elastic.co/u/raghu1)\
**Post date:** [October 25, 2017, 8:32am UTC](https://discuss.elastic.co/t/how-to-remove-a-filed/105173/3 "2017-10-25T08:32:33Z")

</div>

Thankyou magnus  
I tried using mutate { remove\_field =\> ["payload"]}  
And it worked

---

<div class="post-metadata">

**Author:** ![raghu1](https://avatars.discourse-cdn.com/v4/letter/r/ec9cab/32.png) [@raghu1](https://discuss.elastic.co/u/raghu1)\
**Post date:** [October 26, 2017, 4:51am UTC](https://discuss.elastic.co/t/how-to-remove-a-filed/105173/4 "2017-10-26T04:51:02Z")

</div>

Hi Team,

For below input:  
BEGIN|ABC1|3|fdfd||SRServiceRequest|sfdf-20171023-17.02.00.979478|||fdfda|dfd|dfdf||sdfd|fdf|dfgfdkj|fdfdf|AUDIT2|2017-10-23T17:02:01.113+08:00|fdfdf|EOD

i am using grok pattern:  
[%{GREEDYDATA:BEGIN}]|%{GREEDYDATA:service\_id}|%{GREEDYDATA:log\_level}|%{GREEDYDATA:log\_type}|%{GREEDYDATA:log\_code}|%{GREEDYDATA:log\_message}|%{GREEDYDATA:log\_guid}|%{GREEDYDATA:org\_code}|%{GREEDYDATA:process\_name}|%{GREEDYDATA:si\_id}|%{GREEDYDATA:operation\_name}|%{GREEDYDATA:service\_requestor}|%{GREEDYDATA:service\_provider}|%{GREEDYDATA:machine\_name}|%{GREEDYDATA:engine\_name}|%{GREEDYDATA:payload}|%{GREEDYDATA:pre\_step}|%{GREEDYDATA:post\_step}|%{GREEDYDATA:log\_timestamp}|%{GREEDYDATA:si\_id}|%{GREEDYDATA:EOD}

In below output i see service\_id is populating from BEGIN instead of ABC1 alone.

Output below:  
{  
"BEGIN": [  
null  
],  
"service\_id": [  
"BEGIN|ABC1"  
],  
"log\_level": [  
"3"  
],  
.........more.

Please can you suggest how to populate BEING with BEGIN constant instead of null.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 26, 2017, 5:11am UTC](https://discuss.elastic.co/t/how-to-remove-a-filed/105173/5 "2017-10-26T05:11:43Z")

</div>

As I said: That grok expression is awful. That amount of GREEDYDATA patterns is extremely inefficient and could give the wrong results. Just use a csv filter.

Side note: **Always** post configuration and log entries as preformatted text to avoid getting the text damaged by the discussion forum software. In this case it has probably damaged your grok expression so I can't tell what you're using.

---

<div class="post-metadata">

**Author:** ![raghu1](https://avatars.discourse-cdn.com/v4/letter/r/ec9cab/32.png) [@raghu1](https://discuss.elastic.co/u/raghu1)\
**Post date:** [October 26, 2017, 7:52am UTC](https://discuss.elastic.co/t/how-to-remove-a-filed/105173/6 "2017-10-26T07:52:26Z")

</div>

Hi Magnus,

Sorry. Kindly could you suggest pattern so being element is populated instead of null and filed service\_id does not take BEGIN.

INPUT:  
BEGIN|REWARDPOINTSACCOUNT100|3|AUDIT||SRServiceRequest|IS@SG-156-20171023-17.02.00.979478|||d6e66635-8bc1-486c-9fa5-4e3c04f1dcba|retrievePartyRewardSummary|ICAL||s01sbw5app2a|RewardPointsAccount\_SG\_v1\_0\_513\_2-Process\_Archive|\<?xml version="1.0" encoding="UTF-8"?\> \<soap:Envelope xmlns:soap="[http://schemas.xmlsoap.org/soap/envelope/](http://schemas.xmlsoap.org/soap/envelope/)"\>\<soap:Header\>\<ns4:MsgDetl xmlns:ns4="[http://schemas.care.com/soi/common/4\_0](http://schemas.care.com/soi/common/4_0)" xmlns:ns3="[http://schemas.care.com/soi/retrievePartyRewardSummary/4\_0](http://schemas.care.com/soi/retrievePartyRewardSummary/4_0)" xmlns:ns2="[http://schemas.care.com/soi/common/4\_1](http://schemas.care.com/soi/common/4_1)"\>\<ns4:MsgVersion\>4.0\</ns4:MsgVersion\>\<ns4:MsgUID\>IS@SG-156-20171023-17.02.00.979478\</ns4:MsgUID\>\<ns4:SvcVersion\>4.0.0\</ns4:SvcVersion\>\</ns4:MsgDetl\>\<ns4:Trace xmlns:ns4="[http://schemas.care.com/soi/common/4\_0](http://schemas.care.com/soi/common/4_0)" xmlns:ns3="[http://schemas.care.com/soi/retrievePartyRewardSummary/4\_0](http://schemas.care.com/soi/retrievePartyRewardSummary/4_0)" xmlns:ns2="[http://schemas.care.com/soi/common/4\_1](http://schemas.care.com/soi/common/4_1)"\>\<ns4:RqDateTime\>2017-10-23T17:02:00.980+08:00\</ns4:RqDateTime\>\<ns4:RqClient\>\<ns4:RqClientId\>ICAL\</ns4:RqClientId\>\<ns4:RqClientOrg\>0001\</ns4:RqClientOrg\>\<ns4:RqClientCtry\>SG\</ns4:RqClientCtry\>\</ns4:RqClient\>\<ns4:Operator\>\<ns4:OpInternalId\>DRADICA2\</ns4:OpInternalId\>\<ns4:OpLoginId\>CRMUATA\</ns4:OpLoginId\>\<ns4:OpRole\>TELLER\</ns4:OpRole\>\</ns4:Operator\>\</ns4:Trace\>\<ns4:ExtendedHeader xmlns:ns4="[http://schemas.care.com/soi/common/4\_0](http://schemas.care.com/soi/common/4_0)" xmlns:ns3="[http://schemas.care.com/soi/retrievePartyRewardSummary/4\_0](http://schemas.care.com/soi/retrievePartyRewardSummary/4_0)" xmlns:ns2="[http://schemas.care.com/soi/common/4\_1](http://schemas.care.com/soi/common/4_1)"/\>\</soap:Header\>\<soap:Body\>\<ns2:retrievePartyRewardSummary xmlns:ns2="[http://schemas.care.com/soi/retrievePartyRewardSummary/4\_0](http://schemas.care.com/soi/retrievePartyRewardSummary/4_0)" xmlns:ns3="[http://schemas.care.com/soi/common/4\_0](http://schemas.care.com/soi/common/4_0)" xmlns:ns4="[http://schemas.care.com/soi/common/4\_1](http://schemas.care.com/soi/common/4_1)"\>\<ns4:CommonRq\>\<ns4:OrgCode\>0001\</ns4:OrgCode\>\<ns4:ChannelId\>96\</ns4:ChannelId\>\</ns4:CommonRq\>\<ns4:TrailDetl\>\<ns4:OpId\>CRMUATA\</ns4:OpId\>\<ns4:OpDateTime\>2017-10-23T17:02:00.980+08:00\</ns4:OpDateTime\>\<ns4:OpMainUnit\>0117\</ns4:OpMainUnit\>\</ns4:TrailDetl\>\<ns4:CISInternalId\>\<ns4:CISCIN\>S1139026I\</ns4:CISCIN\>\<ns4:CISCINsfx\>00\</ns4:CISCINsfx\>\</ns4:CISInternalId\>\</ns2:retrievePartyRewardSummary\>\</soap:Body\>\</soap:Envelope\>|d6e66635-8bc1-486c-9fa5-4e3c04f1dcba|AUDIT2|2017-10-23T17:02:01.113+08:00|d6e66635-8bc1-486c-9fa5-4e3c04f1dcba|EOD

PATTERN:  
[%{GREEDYDATA:BEGIN}|%{GREEDYDATA:service\_id}|%{GREEDYDATA:log\_level}|%{GREEDYDATA:log\_type}|%{GREEDYDATA:log\_code}|%{GREEDYDATA:log\_message}|%{GREEDYDATA:log\_guid}|%{GREEDYDATA:org\_code}|%{GREEDYDATA:process\_name}|%{GREEDYDATA:si\_id}|%{GREEDYDATA:operation\_name}|%{GREEDYDATA:service\_requestor}|%{GREEDYDATA:service\_provider}|%{GREEDYDATA:machine\_name}|%{GREEDYDATA:engine\_name}|%{GREEDYDATA:pay\_load}|%{GREEDYDATA:pre\_step}|%{GREEDYDATA:post\_step}|%{GREEDYDATA:log\_timestamp}|%{GREEDYDATA:si\_id}|%{GREEDYDATA:EOD}

# OUTPUT:

# { "BEGIN": [null], "service\_id": ["BEGIN|REWARDPOINTSACCOUNT100"], "log\_level": ["3"], "log\_type": ["AUDIT"], "log\_code": [""], "log\_message": ["SRServiceRequest"], "log\_guid": ["IS@SG-156-20171023-17.02.00.979478"], "org\_code": [""], "process\_name": [""], "si\_id": ["d6e66635-8bc1-486c-9fa5-4e3c04f1dcba", "d6e66635-8bc1-486c-9fa5-4e3c04f1dcba"], "operation\_name": ["retrievePartyRewardSummary"], "service\_requestor": ["ICAL"], "service\_provider": [""], "machine\_name": ["s01sbw5app2a"], "engine\_name": ["RewardPointsAccount\_SG\_v1\_0\_513\_2-Process\_Archive"], "pay\_load": ["\<?xml version="1.0" encoding="UTF-8"?\> \<soap:Envelope xmlns:soap="[http://schemas.xmlsoap.org/soap/envelope/](http://schemas.xmlsoap.org/soap/envelope/)"\>\<soap:Header\>\<ns4:MsgDetl xmlns:ns4="[http://schemas.care.com/soi/common/4\_0](http://schemas.care.com/soi/common/4_0)" xmlns:ns3="[http://schemas.care.com/soi/retrievePartyRewardSummary/4\_0](http://schemas.care.com/soi/retrievePartyRewardSummary/4_0)" xmlns:ns2="[http://schemas.care.com/soi/common/4\_1](http://schemas.care.com/soi/common/4_1)"\>\<ns4:MsgVersion\>4.0\</ns4:MsgVersion\>\<ns4:MsgUID\>IS@SG-156-20171023-17.02.00.979478\</ns4:MsgUID\>\<ns4:SvcVersion\>4.0.0\</ns4:SvcVersion\>\</ns4:MsgDetl\>\<ns4:Trace xmlns:ns4="[http://schemas.care.com/soi/common/4\_0](http://schemas.care.com/soi/common/4_0)" xmlns:ns3="[http://schemas.care.com/soi/retrievePartyRewardSummary/4\_0](http://schemas.care.com/soi/retrievePartyRewardSummary/4_0)" xmlns:ns2="[http://schemas.care.com/soi/common/4\_1](http://schemas.care.com/soi/common/4_1)"\>\<ns4:RqDateTime\>2017-10-23T17:02:00.980+08:00\</ns4:RqDateTime\>\<ns4:RqClient\>\<ns4:RqClientId\>ICAL\</ns4:RqClientId\>\<ns4:RqClientOrg\>0001\</ns4:RqClientOrg\>\<ns4:RqClientCtry\>SG\</ns4:RqClientCtry\>\</ns4:RqClient\>\<ns4:Operator\>\<ns4:OpInternalId\>DRADICA2\</ns4:OpInternalId\>\<ns4:OpLoginId\>CRMUATA\</ns4:OpLoginId\>\<ns4:OpRole\>TELLER\</ns4:OpRole\>\</ns4:Operator\>\</ns4:Trace\>\<ns4:ExtendedHeader xmlns:ns4="[http://schemas.care.com/soi/common/4\_0](http://schemas.care.com/soi/common/4_0)" xmlns:ns3="[http://schemas.care.com/soi/retrievePartyRewardSummary/4\_0](http://schemas.care.com/soi/retrievePartyRewardSummary/4_0)" xmlns:ns2="[http://schemas.care.com/soi/common/4\_1](http://schemas.care.com/soi/common/4_1)"/\>\</soap:Header\>\<soap:Body\>\<ns2:retrievePartyRewardSummary xmlns:ns2="[http://schemas.care.com/soi/retrievePartyRewardSummary/4\_0](http://schemas.care.com/soi/retrievePartyRewardSummary/4_0)" xmlns:ns3="[http://schemas.care.com/soi/common/4\_0](http://schemas.care.com/soi/common/4_0)" xmlns:ns4="[http://schemas.care.com/soi/common/4\_1](http://schemas.care.com/soi/common/4_1)"\>\<ns4:CommonRq\>\<ns4:OrgCode\>0001\</ns4:OrgCode\>\<ns4:ChannelId\>96\</ns4:ChannelId\>\</ns4:CommonRq\>\<ns4:TrailDetl\>\<ns4:OpId\>CRMUATA\</ns4:OpId\>\<ns4:OpDateTime\>2017-10-23T17:02:00.980+08:00\</ns4:OpDateTime\>\<ns4:OpMainUnit\>0117\</ns4:OpMainUnit\>\</ns4:TrailDetl\>\<ns4:CISInternalId\>\<ns4:CISCIN\>S1139026I\</ns4:CISCIN\>\<ns4:CISCINsfx\>00\</ns4:CISCINsfx\>\</ns4:CISInternalId\>\</ns2:retrievePartyRewardSummary\>\</soap:Body\>\</soap:Envelope\>" ], "pre\_step": ["d6e66635-8bc1-486c-9fa5-4e3c04f1dcba"], "post\_step": ["AUDIT2"], "log\_timestamp": ["2017-10-23T17:02:01.113+08:00"], "EOD": ["EOD"] }

---

<div class="post-metadata">

**Author:** ![RohanKumbhar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohankumbhar/32/23649_2.png) [@RohanKumbhar](https://discuss.elastic.co/u/RohanKumbhar)\
**Post date:** [October 26, 2017, 8:58am UTC](https://discuss.elastic.co/t/how-to-remove-a-filed/105173/7 "2017-10-26T08:58:30Z")

</div>

> [@raghu1](#):
>
> BEGIN|REWARDPOINTSACCOUNT100|3|AUDIT||SRServiceRequest|IS@SG-156-20171023-17.02.00.979478|||d6e66635-8bc1-486c-9fa5-4e3c04f1dcba|retrievePartyRewardSummary|ICAL||s01sbw5app2a|RewardPointsAccount\_SG\_v1\_0\_513\_2-Process\_Archive|\<?xml version="1.0" encoding="UTF-8"?\> \<soap:Envelope xmlns:soap="[http://schemas.xmlsoap.org/soap/envelope/](http://schemas.xmlsoap.org/soap/envelope/)"\>\<soap:Header\>\<ns4:MsgDetl xmlns:ns4="[http://schemas.care.com/soi/common/4\_0](http://schemas.care.com/soi/common/4_0)" xmlns:ns3="[http://schemas.care.com/soi/retrievePartyRewardSummary/4\_0](http://schemas.care.com/soi/retrievePartyRewardSummary/4_0)" xmlns:ns2="[http://schemas.care.com/soi/common/4\_1](http://schemas.care.com/soi/common/4_1)"\>\<ns4:MsgVersion\>4.0\</ns4:MsgVersion\>\<ns4:MsgUID\>IS@SG-156-20171023-17.02.00.979478\</ns4:MsgUID\>\<ns4:SvcVersion\>4.0.0\</ns4:SvcVersion\>\</ns4:MsgDetl\>\<ns4:Trace xmlns:ns4="[http://schemas.care.com/soi/common/4\_0](http://schemas.care.com/soi/common/4_0)" xmlns:ns3="[http://schemas.care.com/soi/retrievePartyRewardSummary/4\_0](http://schemas.care.com/soi/retrievePartyRewardSummary/4_0)" xmlns:ns2="[http://schemas.care.com/soi/common/4\_1](http://schemas.care.com/soi/common/4_1)"\>\<ns4:RqDateTime\>2017-10-23T17:02:00.980+08:00\</ns4:RqDateTime\>\<ns4:RqClient\>\<ns4:RqClientId\>ICAL\</ns4:RqClientId\>\<ns4:RqClientOrg\>0001\</ns4:RqClientOrg\>\<ns4:RqClientCtry\>SG\</ns4:RqClientCtry\>\</ns4:RqClient\>\<ns4:Operator\>\<ns4:OpInternalId\>DRADICA2\</ns4:OpInternalId\>\<ns4:OpLoginId\>CRMUATA\</ns4:OpLoginId\>\<ns4:OpRole\>TELLER\</ns4:OpRole\>\</ns4:Operator\>\</ns4:Trace\>\<ns4:ExtendedHeader xmlns:ns4="[http://schemas.care.com/soi/common/4\_0](http://schemas.care.com/soi/common/4_0)" xmlns:ns3="[http://schemas.care.com/soi/retrievePartyRewardSummary/4\_0](http://schemas.care.com/soi/retrievePartyRewardSummary/4_0)" xmlns:ns2="[http://schemas.care.com/soi/common/4\_1](http://schemas.care.com/soi/common/4_1)"/\>\</soap:Header\>\<soap:Body\>\<ns2:retrievePartyRewardSummary xmlns:ns2="[http://schemas.care.com/soi/retrievePartyRewardSummary/4\_0](http://schemas.care.com/soi/retrievePartyRewardSummary/4_0)" xmlns:ns3="[http://schemas.care.com/soi/common/4\_0](http://schemas.care.com/soi/common/4_0)" xmlns:ns4="[http://schemas.care.com/soi/common/4\_1](http://schemas.care.com/soi/common/4_1)"\>\<ns4:CommonRq\>\<ns4:OrgCode\>0001\</ns4:OrgCode\>\<ns4:ChannelId\>96\</ns4:ChannelId\>\</ns4:CommonRq\>\<ns4:TrailDetl\>\<ns4:OpId\>CRMUATA\</ns4:OpId\>\<ns4:OpDateTime\>2017-10-23T17:02:00.980+08:00\</ns4:OpDateTime\>\<ns4:OpMainUnit\>0117\</ns4:OpMainUnit\>\</ns4:TrailDetl\>\<ns4:CISInternalId\>\<ns4:CISCIN\>S1139026I\</ns4:CISCIN\>\<ns4:CISCINsfx\>00\</ns4:CISCINsfx\>\</ns4:CISInternalId\>\</ns2:retrievePartyRewardSummary\>\</soap:Body\>\</soap:Envelope\>|d6e66635-8bc1-486c-9fa5-4e3c04f1dcba|AUDIT2|2017-10-23T17:02:01.113+08:00|d6e66635-8bc1-486c-9fa5-4e3c04f1dcba|EOD

You can try with below grok pattern

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/0/a00594fff79425ec80f858ab027f11656d011f2e.png)

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 26, 2017, 10:05am UTC](https://discuss.elastic.co/t/how-to-remove-a-filed/105173/8 "2017-10-26T10:05:25Z")

</div>

> You can try with below grok pattern

This might appear to work but it's still insanely inefficient and potentially fragile.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 23, 2017, 10:05am UTC](https://discuss.elastic.co/t/how-to-remove-a-filed/105173/9 "2017-11-23T10:05:53Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.