Logstash - filebeat

[Arezki76 (Ladjadj)]

Hi,

I'm new in ELK and I'm trying to parse log file as follow:
2019 Nov 12 14:58:01:211 GMT -0500 BW.ypg_bis_ful_campaignservice_bw-ypg-bis-ful-campaignservice-bw Info [BW-User] UPSTREAM_REQUEST Job-1000 [ESBCore/Log/System/Log.process/Log Message]: <?xml version="1.0" encoding="UTF-8"?>
<ns0:LogEntry xmlns:ns0="urn:com.ypg.core.schemas.logging.v2">
<ns1:ContextHeader xmlns:ns1="urn:com.ypg.common.types.v1">
ns1:OriginatorIdWebServices</ns1:OriginatorId>
ns1:SystemApplicationNameypg.bis.ful.campaignservice.v1</ns1:SystemApplicationName>
ns1:SystemServiceNameCampaignService</ns1:SystemServiceName>
ns1:SystemOperationNameListCampaignProducts</ns1:SystemOperationName>
ns1:SystemTrackingId20d3dd47-fca7-4885-926a-e562cbb9fe42</ns1:SystemTrackingId>
ns1:SystemTimeCreated2019-11-12T14:58:01.197-05:00</ns1:SystemTimeCreated>
ns1:TibcoProcessId1000</ns1:TibcoProcessId>
</ns1:ContextHeader>
ns0:LogEntryRequest
ns0:LogLevelINFO</ns0:LogLevel>
ns0:MsgCodeUPSTREAM_REQUEST</ns0:MsgCode>
ns0:MessageRequest received</ns0:Message>
ns0:Payload<?xml version="1.0" encoding="UTF-8"?>
<Payload><urn:ListCampaignProductsRequest xmlns:urn="urn:com.ypg.bis.ful.campaignservice.v1" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:urn1="urn:com.ypg.common.service.types.v1" xmlns:urn2="urn:com.ypg.common.types.v1" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" timezone="+00:00" validateOnly="false"><urn1:ContextHeader><urn2:OriginatorId>WebServices</urn2:OriginatorId></urn1:ContextHeader><urn:MerchantId>623778</urn:MerchantId></urn:ListCampaignProductsRequest></Payload></ns0:Payload>
ns0:Timestamp2019-11-12T14:58:01.206-05:00</ns0:Timestamp>
</ns0:LogEntryRequest>
</ns0:LogEntry>

I want that the below part will be tagged as the message
UPSTREAM_REQUEST Job-1000 [ESBCore/Log/System/Log.process/Log Message]: <?xml version="1.0" encoding="UTF-8"?>
<ns0:LogEntry xmlns:ns0="urn:com.ypg.core.schemas.logging.v2">
<ns1:ContextHeader xmlns:ns1="urn:com.ypg.common.types.v1">
ns1:OriginatorIdWebServices</ns1:OriginatorId>
ns1:SystemApplicationNameypg.bis.ful.campaignservice.v1</ns1:SystemApplicationName>
ns1:SystemServiceNameCampaignService</ns1:SystemServiceName>
ns1:SystemOperationNameListCampaignProducts</ns1:SystemOperationName>
ns1:SystemTrackingId20d3dd47-fca7-4885-926a-e562cbb9fe42</ns1:SystemTrackingId>
ns1:SystemTimeCreated2019-11-12T14:58:01.197-05:00</ns1:SystemTimeCreated>
ns1:TibcoProcessId1000</ns1:TibcoProcessId>
</ns1:ContextHeader>
ns0:LogEntryRequest
ns0:LogLevelINFO</ns0:LogLevel>
ns0:MsgCodeUPSTREAM_REQUEST</ns0:MsgCode>
ns0:MessageRequest received</ns0:Message>
ns0:Payload<?xml version="1.0" encoding="UTF-8"?>
<Payload><urn:ListCampaignProductsRequest xmlns:urn="urn:com.ypg.bis.ful.campaignservice.v1" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:urn1="urn:com.ypg.common.service.types.v1" xmlns:urn2="urn:com.ypg.common.types.v1" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" timezone="+00:00" validateOnly="false"><urn1:ContextHeader><urn2:OriginatorId>WebServices</urn2:OriginatorId></urn1:ContextHeader><urn:MerchantId>623778</urn:MerchantId></urn:ListCampaignProductsRequest></Payload></ns0:Payload>
ns0:Timestamp2019-11-12T14:58:01.206-05:00</ns0:Timestamp>
</ns0:LogEntryRequest>
</ns0:LogEntry>

I used the follwoing regex in my logstash config file, but seems not working:
match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{DATA:TZ} %{DATA:GMT} %{GREEDYDATA:ApplicationName} %{LOGLEVEL:LOG-Level} [%{DATA:Logger}] (?(.|\r|\n)*)"}

Can someone help please ?

Thanks

[Archie_Crawford]

Before I put my grok patterns in logstash I test them in https://grokdebug.herokuapp.com/.. you should be able to trouble shoot whats wrong with you grok pattern there. It looks like your pattern works up to here %{SYSLOGTIMESTAMP:timestamp}%{DATA:TZ} %{DATA:GMT} %{GREEDYDATA:ApplicationName} %{LOGLEVEL:LOG-Level}.. but the rest of it causes the error

[Arezki76 (Ladjadj)]

Thank you @Archie_Crawford,

Yes I tested my pattern in the grokdebug but I don't know why the multiline is bypassed.

Regards.