Is it possible to remove/work http headers?

Elastic Stack Logstash
1

Hi all,

we are investigating and building some proof of concept to validate the usage of ELK

One use case is log events analyze and visualization.

Each log event is one line in database table.
For each event we store the "full" http request, here we understand "full http request" as

"full http request" = https headers + http payload (xml)

some questions:

  1. using logstash filters (mutate ?) how can we discard the http headers block ? The goal is to discard the http headers, so we keep only the xml in the payload. Then we need to extrack information from the xml (generatorEntity, generatoraplication, generatoruser)

  2. on question 1) i asked on the possibility of discarding http headers, here i ask on the possibility of "working" the values from http header and store it in an field (example: store x-remote-server in a field and latter do some analysis like count of requests by remote server in Kibana)

Bellow an example of the "full http request"

 POST http://xxxxx.int:7801/FileNetWS_V3 HTTP/1.1
 Host: xxxx.int:7801
 Accept: application/soap+xml,multipart/related,text/*
 User-Agent: IBM WebServices/1.0
 Cache-Control: no-cache
 Pragma: no-cache
 SOAPAction: "http://xxxxxx.pt/FilenetWS_V3GetDocContent"
 Connection: Keep-Alive
 Content-Type: text/xml; charset=UTF-8
 Content-Length: 1044
 Date: Mon, 11 Jul 2016 10:16:38 GMT
 X-Remote-Addr: 10.99.99.999
 X-Remote-Host: 10.99.99.1999
 X-Server-Name: xxxxxx.int
 X-Server-Port: 7801


     <NS1:GetDocContent xmlns:NS1="http://www.acoreanaseguros.pt/FilenetWS_V3">
    <Header>
        <NS2:Version xmlns:NS2="http://www.acoreanaseguros.pt/AcorGenService">1.0</NS2:Version>
        <NS3:Timestamp xmlns:NS3="http://www.acoreanaseguros.pt/AcorGenService">2016-07-11 11:16:39.605</NS3:Timestamp>
        <NS4:MessageGuid xmlns:NS4="http://www.acoreanaseguros.pt/AcorGenService">d904fdc4-6d9f-490c-92db-98b80e822139</NS4:MessageGuid>
        <NS5:GeneratorEntity xmlns:NS5="http://www.acoreanaseguros.pt/AcorGenService">ACOREANA</NS5:GeneratorEntity>
        <NS6:GeneratorApplication xmlns:NS6="http://www.acoreanaseguros.pt/AcorGenService">SIGA</NS6:GeneratorApplication>
        <NS7:ServiceId xmlns:NS7="http://www.acoreanaseguros.pt/AcorGenService">FilenetWS_V3/GetDocContent</NS7:ServiceId>
    </Header>
    <Body>
        <system_name>xxxxx</system_name>
        <docId>69345903</docId>
        <docPage>0</docPage>
        <conversion>0</conversion>
    </Body>
</NS1:GetDocContent>

Best regards

Rui Madaleno