Remove_fields dynamic

Elastic Stack Logstash
1

Is there a way to use remove_fields as in a dynamic way
for example:

 remove_field => ["message",
                  "[Request][Header]",
			      "[Request][xmlns:xsi]",
			      "[Request][Body][q1:InvoiceQueryOpenRequest][*][xlp:type]"
			  ]

to remove xlp:type within all fields

the parsed xml is this:

<soap:Body soap:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
        <tns:getOpenInvoices>
            <invoiceQueryOpenRequest href="#id1" />
        </tns:getOpenInvoices>
        <q1:InvoiceQueryOpenRequest id="id1" xsi:type="q1:InvoiceQueryOpenRequest" xmlns:q1="java:com.turkcelltech.collgw.model.invoice">
            <bankId xsi:type="xsd:int">23</bankId>
            <compId xsi:type="xsd:int">533</compId>
            <curr xsi:type="xsd:string">949</curr>
            <custId xsi:nil="true" />
            <invCount xsi:type="xsd:int">5</invCount>
            <msgDate xsi:nil="true" />
            <msisdn xsi:type="xsd:long">****************</msisdn>
            <orig xsi:nil="true" />
            <period xsi:type="xsd:string">201901</period>
            <procDate xsi:nil="true" />
            <procTime xsi:nil="true" />
            <sessionId xsi:type="xsd:string">*****************</sessionId>
            <stan xsi:type="xsd:long">0</stan>
        </q1:InvoiceQueryOpenRequest>
    </soap:Body>
2

No, you cannot do that with remove_field. You might be able to remove items from the XML using mutate+gsub filter or from the parsed XML using a ruby filter. That's not the parsed XML, right? It is the XML itself. What does xlp:type refer to?

3

Yes it is the main XML it is parsed to JSON and xlp:type is written wrong it suppose to be xsi:type. So I need to use ruby filter to remove xsi:type within my json file.

My json output:

"theXML":{  
      "Body":{  
         "env:encodingStyle":"http://schemas.xmlsoap.org/soap/encoding/",
         "getOpenInvoicesResponse":{  
            "result":{  
               "msgDate":{  
                  "xsi:type":"xsd:string",
                  "content":"0118160750"
               },
               "xsi:type":"n1:InvoiceQueryOpenResponse",
               "compId":{  
                  "xsi:type":"xsd:int",
                  "content":"533"
               },
               "invCount":{  
                  "xsi:type":"xsd:int",
                  "content":"1"
               },
               "curr":{  
                  "xsi:type":"xsd:string",
                  "content":"949"
               },
               "xmlns:n1":"java:com.turkcelltech.collgw.model.invoice",
               "errMsg":{  
                  "xsi:type":"xsd:string",
                  "content":"ok"
               },
               "invoiceList":{  
                  "InvoiceDetail":{  
                     "xsi:type":"n1:InvoiceDetail",
                     "prevMsisdn":{  
                        "xsi:type":"xsd:long",
                        "content":"***********"
                     },
                     "controlId":{  
                        "xsi:type":"xsd:string",
                        "content":"593045584147"
                     },
                     "invAmount":{  
                        "xsi:type":"xsd:double",
                        "content":"13.85"
                     },
                     "nameSurname":{  
                        "xsi:type":"xsd:string",
                        "content":"La****** Br****"
                     },
                     "remark":{  
                        "xsi:type":"xsd:string"
                     },
                     "invType":{  
                        "xsi:type":"xsd:int",
                        "content":"0"
                     },
                     "invNo":{  
                        "xsi:type":"xsd:long",
                        "content":"21282404"
                     },
                     "duedate":{  
                        "xsi:type":"xsd:string",
                        "content":"20190118"
                     },
                     "orderNo":{  
                        "xsi:type":"xsd:int",
                        "content":"0"
                     },
                     "period":{  
                        "xsi:type":"xsd:string",
                        "content":"201812"
                     }
                  },
                  "soapenc:arrayType":"n1:InvoiceDetail[1]"
               },
               "stan":{  
                  "xsi:type":"xsd:long",
                  "content":"0"
               },
               "msisdn":{  
                  "xsi:type":"xsd:long",
                  "content":"************"
               },
               "orig":{  
                  "xsi:nil":"true"
               },
               "respCode":{  
                  "xsi:type":"xsd:string",
                  "content":"00"
               },
               "bankId":{  
                  "xsi:type":"xsd:int",
                  "content":"23"
               }
            },
            "xmlns:m":"http://my.*******.com.tr:7002/KKTCCollGWws/KKTCCollGW"
         }
      }
   }
4

Why not use mutate+gsub to remove it before parsing?

5

You are right, in my config file I add another filter like this and its worked

filter {
	mutate {
			gsub => ["message", "xsi:\w+=(?<grp1>\"|')\w+(:\w+)?\k<grp1>\s*", ""]
	}
}
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.