# Remove unwanted fields

**URL:** <https://discuss.elastic.co/t/remove-unwanted-fields/178795>\
**Category:** Logstash\
**Created:** [April 28, 2019, 10:26am UTC](https://discuss.elastic.co/t/remove-unwanted-fields/178795 "2019-04-28T10:26:40Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jan\_Kaspar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jan_kaspar/32/44443_2.png) [@Jan\_Kaspar](https://discuss.elastic.co/u/Jan_Kaspar)\
**Post date:** [April 28, 2019, 10:26am UTC](https://discuss.elastic.co/t/remove-unwanted-fields/178795/1 "2019-04-28T10:26:40Z")

</div>

Hi all!,  
I just have a question. I am using filebeat for parsing XML file generated by NPS. It seems like it working fine. There are some things i would like to adjust.

This is message:

04/28/2019 09:44:02.897W2012-ARAS28311 1 fe80::1885:2544:9a38:1d45 04/25/2019 15:36:33 274XXX\john.doeXXX\john.doe1Microsoft Routing and Remote Access Service Policy316

In logstash u have this:

if "nps" in [tags] {  
xml {  
source =\> "message"  
target =\> "theXML"  
force\_array =\> false  
}  
}

it causes that all xml parameters are perfectly recognized. But i got two fields for every value:

theXML.Computer-Name.content  
theXML.Computer-Name.data\_type

I would like to remove all fields ending with **.data\_type**  
How can I do that? I fourn some ruby code, but it didnt worked for me.  
Can someone helped me with that?

Jan

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [April 29, 2019, 1:10pm UTC](https://discuss.elastic.co/t/remove-unwanted-fields/178795/2 "2019-04-29T13:10:28Z")

</div>

I am not sure I undestand you issue. Filebeat cannot parse XML. So I guess the issue is not related to Filebeat, but to Logstash. Could you please provide more context?

---

<div class="post-metadata">

**Author:** ![Jan\_Kaspar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jan_kaspar/32/44443_2.png) [@Jan\_Kaspar](https://discuss.elastic.co/u/Jan_Kaspar)\
**Post date:** [April 29, 2019, 1:30pm UTC](https://discuss.elastic.co/t/remove-unwanted-fields/178795/3 "2019-04-29T13:30:30Z")

</div>

> [@kvch](#):
>
> am not sure I undesta

I am able to parse all fields. My issue is that i have a lot of values.

Actualy every field is parsed in to the two values.

theXML.Computer-Name.content  
theXML.Computer-Name.data\_type  
theXML.User-Name.content  
theXML.User-Name.data\_type

If i have 40 field, then i got 80 fields. How can i remove all fields end by data\_type.

Jan

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [April 30, 2019, 7:14am UTC](https://discuss.elastic.co/t/remove-unwanted-fields/178795/4 "2019-04-30T07:14:43Z")

</div>

Unfortunately, Filebeat is not able to drop fields based on endings of field names. It can drop only exactly matching field names.

---

<div class="post-metadata">

**Author:** ![Jan\_Kaspar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jan_kaspar/32/44443_2.png) [@Jan\_Kaspar](https://discuss.elastic.co/u/Jan_Kaspar)\
**Post date:** [April 30, 2019, 7:54am UTC](https://discuss.elastic.co/t/remove-unwanted-fields/178795/5 "2019-04-30T07:54:53Z")

</div>

Oh i wanted to do that at a logstash level... ☹

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [April 30, 2019, 10:03am UTC](https://discuss.elastic.co/t/remove-unwanted-fields/178795/6 "2019-04-30T10:03:46Z")

</div>

Then I am moving you question to Logstash forum.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 30, 2019, 2:06pm UTC](https://discuss.elastic.co/t/remove-unwanted-fields/178795/7 "2019-04-30T14:06:50Z")

</div>

If you use

```
output { stdout { codec => rubydebug } }

```

what does Computer-Name look like? Is it a hash containing content and data\_type objects, or do you have periods in your event names?

---

<div class="post-metadata">

**Author:** ![Jan\_Kaspar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jan_kaspar/32/44443_2.png) [@Jan\_Kaspar](https://discuss.elastic.co/u/Jan_Kaspar)\
**Post date:** [May 1, 2019, 8:55am UTC](https://discuss.elastic.co/t/remove-unwanted-fields/178795/8 "2019-05-01T08:55:52Z")

</div>

Hi,

I dont know why you asking this, I will check it and let you know. But what i need is just to remove fields based on thei names.

I have source gathered by filebeat:

\<Event\>\<Timestamp data\_type="4"\>04/28/2019 13:10:59.245\</Timestamp\>\<Computer-Name data\_type="1"\>W2012-A\</Computer-Name\>\<Event-Source data\_type="1"\>RAS\</Event-Source\>\<NAS-Identifier data\_type="1"\>W2012-A\</NAS-Identifier\>\<NAS-IP-Address data\_type="3"\>192.168.4.221\</NAS-IP-Address\>\<Service-Type data\_type="0"\>2\</Service-Type\>\<Framed-Protocol data\_type="0"\>1\</Framed-Protocol\>\<NAS-Port data\_type="0"\>129\</NAS-Port\>\<NAS-Port-Type data\_type="0"\>5\</NAS-Port-Type\>\<Tunnel-Type data\_type="0"\>1\</Tunnel-Type\>\<Tunnel-Medium-Type data\_type="0"\>1\</Tunnel-Medium-Type\>\<Called-Station-Id data\_type="1"\>192.168.4.221\</Called-Station-Id\>\<Tunnel-Server-Endpt data\_type="1"\>192.168.4.221\</Tunnel-Server-Endpt\>\<Calling-Station-Id data\_type="1"\>192.168.1.221\</Calling-Station-Id\>\<Tunnel-Client-Endpt data\_type="1"\>192.168.1.221\</Tunnel-Client-Endpt\>\<Class data\_type="1"\>311 1 fe80::1885:2544:9a38:1d45 04/25/2019 15:36:33 30\</Class\>\<Acct-Session-Id data\_type="1"\>31\</Acct-Session-Id\>\<User-Name data\_type="1"\>XXX\john.doe\</User-Name\>\<Framed-IP-Address data\_type="3"\>192.168.4.20\</Framed-IP-Address\>\<Framed-MTU data\_type="0"\>1400\</Framed-MTU\>\<Acct-Multi-Session-Id data\_type="1"\>75\</Acct-Multi-Session-Id\>\<Acct-Link-Count data\_type="0"\>1\</Acct-Link-Count\>\<Event-Timestamp data\_type="0"\>1556449857\</Event-Timestamp\>\<Acct-Authentic data\_type="0"\>3\</Acct-Authentic\>\<Acct-Session-Time data\_type="0"\>6\</Acct-Session-Time\>\<Acct-Output-Octets data\_type="0"\>243986\</Acct-Output-Octets\>\<Acct-Input-Octets data\_type="0"\>96667\</Acct-Input-Octets\>\<Acct-Output-Packets data\_type="0"\>429\</Acct-Output-Packets\>\<Acct-Input-Packets data\_type="0"\>436\</Acct-Input-Packets\>\<Acct-Terminate-Cause data\_type="0"\>1\</Acct-Terminate-Cause\>\<Acct-Status-Type data\_type="0"\>2\</Acct-Status-Type\>\<Client-IP-Address data\_type="3"\>192.168.4.221\</Client-IP-Address\>\<Client-Friendly-Name data\_type="1"\>W2012-A\</Client-Friendly-Name\>\<MS-RAS-Vendor data\_type="0"\>311\</MS-RAS-Vendor\>\<MS-RAS-Version data\_type="1"\>MSRASV5.20\</MS-RAS-Version\>\<MS-RAS-Correlation-ID data\_type="1"\>{E5A50A17-DF8D-4333-91E8-F767B7A5F35A}\</MS-RAS-Correlation-ID\>\<MS-RAS-Client-Version data\_type="1"\>MSRASV5.20\</MS-RAS-Client-Version\>\<MS-RAS-Client-Name data\_type="1"\>MSRAS-0-TITAN\</MS-RAS-Client-Name\>\<MS-Network-Access-Server-Type data\_type="0"\>2\</MS-Network-Access-Server-Type\>\<MS-CHAP-Domain data\_type="2"\>01424C554544\</MS-CHAP-Domain\>\<MS-MPPE-Encryption-Types data\_type="0"\>4\</MS-MPPE-Encryption-Types\>\<Proxy-Policy-Name data\_type="1"\>Microsoft Routing and Remote Access Service Policy\</Proxy-Policy-Name\>\<Provider-Type data\_type="0"\>1\</Provider-Type\>\<Packet-Type data\_type="0"\>4\</Packet-Type\>\<Reason-Code data\_type="0"\>0\</Reason-Code\>\</Event\>

You see that every field have also data\_type, that causes that every xml value is parsed into two fields in logstash.

For Example:  
\<Computer-Name data\_type="1"\>W2012-A\</Computer-Name\>

Parsed into:

theXML.Computer-Name.content W2012-A  
theXML.Computer-Name.data\_type 1

I want drop all fields that ends with data\_type.

theXML.Computer-Name.data\_type  
theXML.Client-IP-Address.data\_type  
theXML.Client-Friendly-Name.data\_type  
theXML.Class.data\_type  
theXML.Acct-Terminate-Cause.data\_type  
.  
.  
.

I want drop all of them and same time i dont want name them all manually.

Jan

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 1, 2019, 1:30pm UTC](https://discuss.elastic.co/t/remove-unwanted-fields/178795/9 "2019-05-01T13:30:45Z")

</div>

> [@Jan\_Kaspar](#):
>
> I dont know why you asking this

In logstash, a field within an object is referred to as [objectName][fieldName]. In Kibana I believe that would show up as objectName.fieldName

logstash can also support periods in field names, so you could have a field called objectName.fieldname, but objectName.fieldname and [objectName][fieldName] are very different things.

A xml filter configured as

```
xml { source => "message" target => "theXML" }

```

will produce

```
                    "MS-RAS-Vendor" => [
        [0] {
            "data_type" => "0",
              "content" => "311"
        }
    ],
                   "NAS-IP-Address" => [
        [0] {
            "data_type" => "3",
              "content" => "192.168.4.221"
        }
    ],

```

etc. Do you want that to be reduced to

```
                    "MS-RAS-Vendor" => "311",
                   "NAS-IP-Address" => "192.168.4.221",

```

and so on, or do you want to retain the 'content =\>' part? Assuming you do not want content then this

```
    xml { source => "message" target => "[@metadata][theXML]" remove_field => "message" }
    ruby {
        code => '
            event.get("[@metadata][theXML]").each { |k, v|
                if v.kind_of?(Array)
                    event.set(k, v[0]["content"])
                end
            }
        '
    }

```

should work.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 29, 2019, 1:30pm UTC](https://discuss.elastic.co/t/remove-unwanted-fields/178795/10 "2019-05-29T13:30:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.