Date filter is not working in Logstash

[Harsh_Sharma]

Hi There,

Below is my log.
2018-07-09 08:08:04,689 INFO c.t.t.r.s.i.BookingPaxReportServiceImpl.lambda$generateAndSavePNRReport$1 55d49113-006e-4543-a7e3-2aaa3e757bc1 127.0.0.1 zoomair.uat.i-tare.com http-nio-8080-exec-10 pnrNo=GJ2ZUJ, pnrBkdDateTime=2018-07-05T05:22:55.181, pnrChannel=WEB

I'm trying to apply date filter on 2018-07-09 08:08:04,689

Below is my grok & date filter -

grok {
match => { "message" => "%{DATA:logDateTime}\ %{DATA:LogLevel}\ %{DATA:msg1}\ %{DATA:requestTrackID}\ %{IP:sourceIP}\ %{DATA:tenant}\ %{DATA:thread}\ pnrNo=%{DATA:pnrNo}, pnrBkdDateTime=%{TIMESTAMP_ISO8601:pnrBkdDateTime}, pnrChannel=%{DATA:pnrChannel}" }

date {
match => [ "logDateTime", "yyyy-MM-dd HH:mm:ss" ]
}

All the values are showing on kibana but field 'logDateTime' is not providing expected result. PFA of kibana screenshot.

in message field I'm getting correct timestamp but in logDateTime field (date is correct but time it always showing 05:30:00.000) its not.

Tried a lot but not getting any success. Please help.

Thanks.

[Harsh_Sharma]

I have tried below by making changes in date filter but no luck -

date {
match => [ "logDateTime", "yyyy-MM-dd HH:mm:ss,SSS" ]
target => "read_timestamp"
}

[magnusbaeck (Magnus Bäck)]

What's the contents of the read_timestamp field?

Please don't post screenshots. Copy/paste the JSON text from Kibana's JSON tab.

[Harsh_Sharma]

Hi Magnus,

I read it on some blog so just tried it. Please check Kibana Json. Time value is not there "logDateTime": "2018-07-13",

{
"_index": "kkpnrtest-log-2018.07.14",
"_type": "doc",
"_id": "kpMqmmQBh9hXbMeahaFc",
"_version": 1,
"_score": null,
"_source": {
"pnrPaxCount": 1,
"sectorDcsStatus": "INITIATED",
"pricingRefundAmount": 0,
"paxType": "ADULT",
"pricingTotBaggage": 0,
"sourceIP": "127.0.0.1",
"pricingTotAncillary": 0,
"paxSeatNo": "19C",
"sectorBookingStatus": "INITIATED",
"tenant": "zoomair.uat.i-tare.com",
"pnrCity": "null",
"paxTitle": "Mr",
"type": "log",
"pnrOndOrg": "BLR",
"paxLtvId": "null",
"tags": [
"qp",
"_dateparsefailure"
],
"sectorDest": "BOM",
"pnrCountry": "null",
"pnrReqEmailId": "tkoratha@gmail.com",
"pnrCurrentStatus": "INITIATED",
"pnrChannel": "WEB",
"paxIpAddress": "null",
"pricingTotFees": 150,
"path": "/data/log/newadmin.log",
"pnrCountAdult": 0,
"pricingTotLtvDiscount": 0,
"thread": "http-nio-8080-exec-10",
"paxName": "Thomas Korath",
"requestTrackID": "c.t.t.r.s.i.BookingPaxReportServiceImpl.lambda 55d49113-006e-4543-a7e3-2aaa3e757bc1",
"msg1": "INFO",
"sectorSeqNo": "1",
"pnrSelectedCurrency": "INR",
"pricingExtraServiceFee": 0,
"pricingTotMeal": 0,
"dcsPaymentStatus": 0,
"pricingTotModCharge": 0,
"paxCategory": "null",
"pricingTotDcsMeals": 0,
"paxAddress": "null",
"sectorFltStop": 0,
"pnrBkdDateTime": "2018-07-05T05:22:55.181",
"pnrPaymentStatus": "INITIATED",
"pnrExchangeRate": 1,
"sectorDistance": 925,
"sectorLogicalClass": "B",
"pricingTotSeat": 0,
"pricingTotVendors": 0,
"child": {
" subCategory": [
"Seat",
"BASEPRICE",
"SURCHARGE",
"TAXES",
"FEES"
],
" template": [
"A320-TYPE-A-ECONOMY-B-DFLT",
"-",
"-",
"-",
"-"
],
" cost": [
"0.0",
"4000.0",
"180.0",
"720.0",
"150.0"
],
" {category": [
"FARE",
"FARE",
"TAXESFEES",
"TAXESFEES"
],
" source": [
"BOOKING}",
"BOOKING}",
"BOOKING}",
"BOOKING}",
"BOOKING}"
],
"{category": "ANCILLARY",
" itemcode": [
"19C",
"-",
"-",
"-",
"-"
]
},
"LogLevel": "10:10:10,689",
"sectorCabinClass": "ECONOMY",
"pnrBaseCurrency": "INR",
"pnrPaidAmount": 0,
"pnrReqLName": "Korath",
"pricingTotBasefare": 4000,
"paxNationality": "null",
"pnrTerritory": "null",
"paxLtvAvailablePoints": 0,
"sectorFltNo": "ZA111",
"sectorDepDateTime": "2018-07-26T22:30",
"pricingNetCanCharge": 0,
"paxLtvNetPoints": 0,
"pricingNetAmount": 5050,
"sectorInstanceId": "c59e5d42645a942001645edf871e179b",
"pricingTotTaxes": 720,
"host": "ubuntu-s-1vcpu-2gb-itare",
"pricingTotDiscount": 0,
"pricingTotAdditionalFees": 0,
"sectorRbd": "BS1",
"pnrNo": "GJ2ZUJ",
"pnrFareBasis": "DFLT",
"paxLtvRedeemedPoints": 0,
"pnrRegion": "null",
"sectorOrg": "BLR",
"pnrReqFName": "Thomas",
"message": "2018-07-13 10:10:10,689 INFO c.t.t.r.s.i.BookingPaxReportServiceImpl.lambda 55d49113-006e-4543-a7e3-2aaa3e757bc1 127.0.0.1 zoomair.uat.i-tare.com http-nio-8080-exec-10 pnrNo=GJ2ZUJ, pnrBkdDateTime=2018-07-05T05:22:55.181, pnrChannel=WEB, pnrOndOrg=BLR, pnrOndDest=BOM, pnrAgent=, pnrStaff=, pnrRegion=null, pnrCountry=null, pnrTerritory=null, pnrCity=null, pnrFareBasis=DFLT, pnrBaseCurrency=INR, pnrPaxCount=1, pnrExchangeRate=1.0, pnrMasterAgent=null, pnrSelectedCurrency=INR, pnrPaymentStatus=INITIATED, pnrCurrentStatus=INITIATED, pricingTotDcsBaggage=0.0, pricingTotDcsServices=0.0, dcsPaymentStatus=null, onDPriceDetails= {category=ANCILLARY, subCategory=Seat, template=A320-TYPE-A-ECONOMY-B-DFLT, itemcode=19C, cost=0.0, source=BOOKING}, {category=FARE, subCategory=BASEPRICE, template=-, itemcode=-, cost=4000.0, source=BOOKING}, {category=FARE, subCategory=SURCHARGE, template=-, itemcode=-, cost=180.0, source=BOOKING}, {category=TAXESFEES, subCategory=TAXES, template=-, itemcode=-, cost=720.0, source=BOOKING}, {category=TAXESFEES, subCategory=FEES, template=-, itemcode=-, cost=150.0, source=BOOKING}, pnrHierId=",
"onDPriceDetails": "{category=ANCILLARY, subCategory=Seat, template=A320-TYPE-A-ECONOMY-B-DFLT, itemcode=19C, cost=0.0, source=BOOKING}, {category=FARE, subCategory=BASEPRICE, template=-, itemcode=-, cost=4000.0, source=BOOKING}, {category=FARE, subCategory=SURCHARGE, template=-, itemcode=-, cost=180.0, source=BOOKING}, {category=TAXESFEES, subCategory=TAXES, template=-, itemcode=-, cost=720.0, source=BOOKING}, {category=TAXESFEES, subCategory=FEES, template=-, itemcode=-, cost=150.0, source=BOOKING}",
"pnrPendingAomunt": 5050,
"sectorArrivalDateTime": "2018-07-27T00:00",
"paxTicketNo": "ZA-549-9438-194619",
"paxTvlDirection": "ONWARD",
"@version": "1",
"logDateTime": "2018-07-13",
"pnrReqAddress": "null",
"pnrOndDest": "BOM",
"paxEmailId": "tkoratha@gmail.com",
"@timestamp": "2018-07-14T19:00:35.175Z",
"pricingTotDcsServices": 0
},
"fields": {
"sectorArrivalDateTime": [
"2018-07-27T00:00:00.000Z"
],
"@timestamp": [
"2018-07-14T19:00:35.175Z"
],
"pnrBkdDateTime": [
"2018-07-05T05:22:55.181Z"
],
"sectorDepDateTime": [
"2018-07-26T22:30:00.000Z"
],
"logDateTime": [
"2018-07-13T00:00:00.000Z"
]
},
"sort": [
1531594835175
]
}

[magnusbaeck (Magnus Bäck)]

That's because of a poorly chosen grok expression. Unless you know what you're doing never use more than one DATA or GREEDYDATA pattern in the same grok expression. It's inefficient and can easily result in ambiguous matches like in your case.

To start with, use TIMESTAMP_ISO8601 to match the timestamp. NOTSPACE is another useful pattern that extracts everything up to the next whitespace character.

[Harsh_Sharma]

@Harsh_Sharma

_dateparsefailure

Thanks for the suggestion. I'll keep in mind for future.
Yes, Now its working

Thanks.