How to match more formats by dissect

Elastic Stack Logstash
1

Hi All,

i am having different log formats and i matched one log format by using the dissect combination.

dissect {
mapping => {
"message" => "%{timestamp}|%{component}|%{LogUtils}|%{javaclass}|%{method}|%{delestlog}|%{message_log}|responseTime:%{responsetime}"
}
}

matched Log:
2018-04-10 14:28:07,532|sample-backend|com.example.commerce.backend.utils.LogUtils|com.example.commerce.backend.utils.LogUtils.elastic.CacheCarrierDeliveryEstimatesSearchRepository|indexData()|DELEST_LOG_MSG|RESPONSE_TIME_FROM_ELASTIC_INDEX_DATA|responseTime:8

Need to match for this log :

2018-03-10 14:28:06,182|sample-log|com.example.commerce.backend.utils.LogUtils|com.example.commerce.backend.utils.LogUtils.DeliveryEstimateBusinsessServiceImpl|checkDeliveryEstimates()|DELEST_LOG_MSG|INV_ACTIVE_LOCATIONS|tenantId:fe84657fcd434f0c8eda7eef0afb0dee|internalRequestUUID:3a14fdf9-fee6-4090-8c02-86b57471fc9d|webId:16500|activeLocationIds:[DC-DC_198, DC-DC_396, DC-DC_600, DC-DC_810]

2018-03-10 14:28:06,189|sample-log|com.example.commerce.backend.utils.LogUtils|com.example.commerce.backend.utils.LogUtils.UPSConnector|checkDeliveryEstimates()|DELEST_LOG_MSG|DELREQ_TO_CARRIER|tenantId:fe84657fcd434f0c8eda7eef0afb0dee|internalRequestUUID:3a14fdf9-fee6-4090-8c02-86b57471fc9d|webId:16500|buyerZipCode:01801|shipperZipCode:30517|carrierName:UPS|request:UPSRequest [upsSecurity=UPSSecurity [usernameToken=UsernameToken [username=@gmail.com, password=], serviceAccessToken=ServiceAccessToken [accessLicenseNumber=4D29D430C597E41E]], rateRequest=RateRequest [Request=Request [requestOption=shoptimeintransit, transactionReference=TransactionReference [customerContext=commerce]], shipment=Shipment [deliveryTimeInformation=DeliveryTimeInformation [packageBillType=07, pickup=Pickup [date=20180310, time=1830]], shipper=Shipper [name=commerce, shipperNumber=7V4665, address=Address [postalCode=30517, countryCode=US]], shipTo=ShipTo [name=, address=Address [postalCode=01801, countryCode=US]], shipFrom=ShipFrom [address=Address [postalCode=30517, countryCode=US]], service=Service [code=1DA, description=UPS Next Day Air], pack=[Package [packagingType=PackagingType [code=02, description=Package], dimensions=Dimensions [length=10.0, width=10.0, height=10.0, unitOfMeasurement=UnitOfMeasurement [code=IN, description=Inches]], packageWeight=PackageWeight [unitOfMeasurement=UnitOfMeasurement [code=LBS, description=pounds], weight=20.0]]]]]]

can i extend the exisitng patterns to match to this logs ??

2

As all the logs seem to start with the same fields, use dissect to capture these and store the remaining log data in a single field. You can then use conditionals based on the already extracted fields to select the best way to parse the rest of the data using appropriate combinations of filters.

4

I do not have time to create a full working config for you, but try breaking up the processing into steps like this (I have not tested this):

dissect {
  mapping => {
    "message" => "%{timestamp}|%{component}|%{LogUtils}|%{javaclass}|%{method}|%{delestlog}|%{message_log}|%{recordtype}|%{rest}"
  }
}

if [recordtype] == "RESPONSE_TIME_FROM_ELASTIC_INDEX_DATA" or [recordtype] == "INV_ACTIVE_LOCATIONS" {
  kv {
    field_split => "|"
    value_split => ":"
  }
} else if [recordtype] == "DELREQ_TO_CARRIER" {
  ...
} else {
  ...
}
1 Like
5

Thanks for the Reply. i will try the possibility

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.