Remove unwanted fields

Jan_Kaspar · April 28, 2019, 10:26am

Hi all!,
I just have a question. I am using filebeat for parsing XML file generated by NPS. It seems like it working fine. There are some things i would like to adjust.

This is message:

04/28/2019 09:44:02.897W2012-ARAS28311 1 fe80::1885:2544:9a38:1d45 04/25/2019 15:36:33 274XXX\john.doeXXX\john.doe1Microsoft Routing and Remote Access Service Policy316

In logstash u have this:

if "nps" in [tags] {
xml {
source => "message"
target => "theXML"
force_array => false
}
}

it causes that all xml parameters are perfectly recognized. But i got two fields for every value:

theXML.Computer-Name.content
theXML.Computer-Name.data_type

I would like to remove all fields ending with .data_type
How can I do that? I fourn some ruby code, but it didnt worked for me.
Can someone helped me with that?

Jan

kvch · April 29, 2019, 1:10pm

I am not sure I undestand you issue. Filebeat cannot parse XML. So I guess the issue is not related to Filebeat, but to Logstash. Could you please provide more context?

Jan_Kaspar · April 29, 2019, 1:30pm

I am able to parse all fields. My issue is that i have a lot of values.

Actualy every field is parsed in to the two values.

theXML.Computer-Name.content
theXML.Computer-Name.data_type
theXML.User-Name.content
theXML.User-Name.data_type

If i have 40 field, then i got 80 fields. How can i remove all fields end by data_type.

Jan

kvch · April 30, 2019, 7:14am

Unfortunately, Filebeat is not able to drop fields based on endings of field names. It can drop only exactly matching field names.

Jan_Kaspar · April 30, 2019, 7:54am

Oh i wanted to do that at a logstash level...

kvch · April 30, 2019, 10:03am

Then I am moving you question to Logstash forum.

Badger · April 30, 2019, 2:06pm

If you use

output { stdout { codec => rubydebug } }

what does Computer-Name look like? Is it a hash containing content and data_type objects, or do you have periods in your event names?

Jan_Kaspar · May 1, 2019, 8:55am

Hi,

I dont know why you asking this, I will check it and let you know. But what i need is just to remove fields based on thei names.

I have source gathered by filebeat:

<Event><Timestamp data_type="4">04/28/2019 13:10:59.245</Timestamp><Computer-Name data_type="1">W2012-A</Computer-Name><Event-Source data_type="1">RAS</Event-Source><NAS-Identifier data_type="1">W2012-A</NAS-Identifier><NAS-IP-Address data_type="3">192.168.4.221</NAS-IP-Address><Service-Type data_type="0">2</Service-Type><Framed-Protocol data_type="0">1</Framed-Protocol><NAS-Port data_type="0">129</NAS-Port><NAS-Port-Type data_type="0">5</NAS-Port-Type><Tunnel-Type data_type="0">1</Tunnel-Type><Tunnel-Medium-Type data_type="0">1</Tunnel-Medium-Type><Called-Station-Id data_type="1">192.168.4.221</Called-Station-Id><Tunnel-Server-Endpt data_type="1">192.168.4.221</Tunnel-Server-Endpt><Calling-Station-Id data_type="1">192.168.1.221</Calling-Station-Id><Tunnel-Client-Endpt data_type="1">192.168.1.221</Tunnel-Client-Endpt><Class data_type="1">311 1 fe80::1885:2544:9a38:1d45 04/25/2019 15:36:33 30</Class><Acct-Session-Id data_type="1">31</Acct-Session-Id><User-Name data_type="1">XXX\john.doe</User-Name><Framed-IP-Address data_type="3">192.168.4.20</Framed-IP-Address><Framed-MTU data_type="0">1400</Framed-MTU><Acct-Multi-Session-Id data_type="1">75</Acct-Multi-Session-Id><Acct-Link-Count data_type="0">1</Acct-Link-Count><Event-Timestamp data_type="0">1556449857</Event-Timestamp><Acct-Authentic data_type="0">3</Acct-Authentic><Acct-Session-Time data_type="0">6</Acct-Session-Time><Acct-Output-Octets data_type="0">243986</Acct-Output-Octets><Acct-Input-Octets data_type="0">96667</Acct-Input-Octets><Acct-Output-Packets data_type="0">429</Acct-Output-Packets><Acct-Input-Packets data_type="0">436</Acct-Input-Packets><Acct-Terminate-Cause data_type="0">1</Acct-Terminate-Cause><Acct-Status-Type data_type="0">2</Acct-Status-Type><Client-IP-Address data_type="3">192.168.4.221</Client-IP-Address><Client-Friendly-Name data_type="1">W2012-A</Client-Friendly-Name><MS-RAS-Vendor data_type="0">311</MS-RAS-Vendor><MS-RAS-Version data_type="1">MSRASV5.20</MS-RAS-Version><MS-RAS-Correlation-ID data_type="1">{E5A50A17-DF8D-4333-91E8-F767B7A5F35A}</MS-RAS-Correlation-ID><MS-RAS-Client-Version data_type="1">MSRASV5.20</MS-RAS-Client-Version><MS-RAS-Client-Name data_type="1">MSRAS-0-TITAN</MS-RAS-Client-Name><MS-Network-Access-Server-Type data_type="0">2</MS-Network-Access-Server-Type><MS-CHAP-Domain data_type="2">01424C554544</MS-CHAP-Domain><MS-MPPE-Encryption-Types data_type="0">4</MS-MPPE-Encryption-Types><Proxy-Policy-Name data_type="1">Microsoft Routing and Remote Access Service Policy</Proxy-Policy-Name><Provider-Type data_type="0">1</Provider-Type><Packet-Type data_type="0">4</Packet-Type><Reason-Code data_type="0">0</Reason-Code></Event>

You see that every field have also data_type, that causes that every xml value is parsed into two fields in logstash.

For Example:
<Computer-Name data_type="1">W2012-A</Computer-Name>

Parsed into:

theXML.Computer-Name.content W2012-A
theXML.Computer-Name.data_type 1

I want drop all fields that ends with data_type.

theXML.Computer-Name.data_type
theXML.Client-IP-Address.data_type
theXML.Client-Friendly-Name.data_type
theXML.Class.data_type
theXML.Acct-Terminate-Cause.data_type
.
.
.

I want drop all of them and same time i dont want name them all manually.

Jan

Badger · May 1, 2019, 1:30pm

In logstash, a field within an object is referred to as [objectName][fieldName]. In Kibana I believe that would show up as objectName.fieldName

logstash can also support periods in field names, so you could have a field called objectName.fieldname, but objectName.fieldname and [objectName][fieldName] are very different things.

A xml filter configured as

xml { source => "message" target => "theXML" }

will produce

                    "MS-RAS-Vendor" => [
        [0] {
            "data_type" => "0",
              "content" => "311"
        }
    ],
                   "NAS-IP-Address" => [
        [0] {
            "data_type" => "3",
              "content" => "192.168.4.221"
        }
    ],

etc. Do you want that to be reduced to

                    "MS-RAS-Vendor" =>  "311",
                   "NAS-IP-Address" => "192.168.4.221",

and so on, or do you want to retain the 'content =>' part? Assuming you do not want content then this

    xml { source => "message" target => "[@metadata][theXML]" remove_field => "message" }
    ruby {
        code => '
            event.get("[@metadata][theXML]").each { |k, v|
                if v.kind_of?(Array)
                    event.set(k, v[0]["content"])
                end
            }
        '
    }

should work.

system · May 29, 2019, 1:30pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.