I am getting a dateparsefailure when trying to match the @timestamp with the UNIX_MS date. I tried it as a separate field and that fails too -
Below is the filter -
filter {
xml {
source => "message"
store_xml => false
target => "httpSample"
xpath => [
"/httpSample/@ts", "time",
"/httpSample/@lb", "label",
"/httpSample/@rc", "response_code",
"/httpSample/@rm", "response_message",
"/httpSample/responseHeader/text()", "response_header",
"/httpSample/requestHeader/text()","request_header",
"/httpSample/responseData/text()","response_data",
"/httpSample/java.net.URL/text()","request_url",
"/httpSample/method/text()","method"
]
}
mutate {
convert => ["time", "integer"]
}
date {
match => ["time", "UNIX_MS"]
target => "@timestamp"
}
}
The input sample is as below -
<httpSample t="33149" lt="33149" ts="1701680045582" s="false" lb="Schedule Message" rc="400" rm="Bad Request" tn="Main Thread 1-1" dt="text" de="UTF-8" by="628" ng="1" na="1">
<responseHeader class="java.lang.String">HTTP/1.1 400 Bad Request
x-frame-options: SAMEORIGIN
x-ua-compatible: IE=edge
content-type: application/json;charset=UTF-8
transfer-encoding: chunked
date: Mon, 04 Dec 2023 08:54:38 GMT
server: Server
strict-transport-security: max-age=31536000; includeSubDomains; preload
</responseHeader>
<requestHeader class="java.lang.String">Connection: keep-alive
X-OUID: 9537f12e-13f8-478b-b7d9-3aad6c2e4eb9
Content-Type: application/json
Authorization: Bearer eyJraWQiOiJ5TDBMYkxlWmxaVjl5SnhzTUJLUVlmaWdwWUItRVJ5aTlRMUZiOUUwS09BIiwiYWxnIjoiUlMyNTYifQ.eyJ2ZXIiOjEsImp0aSI6IkFULkZ4UWhfYUdFWHg4ZnU5WHRGcFZ2MnVndThnMHNRTVBiNHZBUUhwVkJlQ00ub2FyMmY1Y2kzNzg0eFE4cnYwaDciLCJpc3MiOiJodHRwczovL3ByZXZpZXcucGVvcGxlY2xvdWQuZXBzaWxvbi5jb20vb2F1dGgyL2F1c3dlNXA3dXhtT3dSRGlIMGg3IiwiYXVkIjoiaGFybW9ueSIsImlhdCI6MTcwMTY4MDA0NSwiZXhwIjoxNzAxNjgzNjQ1LCJjaWQiOiJQQ00tU1RHX1NWTl9PUEVOQU0tTEVHQUNZX0hBUk1PTllfSEFSTU9OWUxJVkUiLCJ1aWQiOiIwMHV5emc3djlsTE5aZUNNZTBoNyIsInNjcCI6WyJvZmZsaW5lX2FjY2VzcyIsIm9wZW5hbUxlZ2FjeS5naXZlbm5hbWUiLCJvcGVuYW1MZWdhY3kuZW1wbG95ZWVOdW1iZXIiLCJvcGVuYW1MZWdhY3kuY24iLCJvcGVuYW1MZWdhY3kuc24iLCJvcGVuYW1MZWdhY3kudWlkIiwib3BlbmFtTGVnYWN5Lm1haWwiXSwiYXV0aF90aW1lIjoxNzAxNjgwMDQ1LCJvcGVuYW1MZWdhY3kubWFpbCI6Ikhhcm1vbnlfUGVyZm9ybWFuY2VfVGVhbUBlcHNpbG9uLmNvbSIsInN1YiI6ImptZXRlckFQSUBzdGcuZXBzaWxvbi5zZXJ2aWNlLmFjY291bnRzIiwib3BlbmFtTGVnYWN5LnN0Z1BDTVVzZXJJZCI6ImYyYWRkZTcxLWNjMzUtNGM3Yi1iMjU1LTEzM2Y1MmQ1MzIxMyIsIm9wZW5hbUxlZ2FjeS5jbiI6IlBlcmZxYSBwZXJmcWEiLCJvcGVuYW1MZWdhY3kuc24iOiJwZXJmcWEiLCJvcGVuYW1MZWdhY3kuZ2l2ZW5uYW1lIjoiUGVyZnFhIiwib3BlbmFtTGVnYWN5LnVpZCI6ImptZXRlckFQSSJ9.bXBOFWQdfQbXLIg_A3BFXYkdQkDLY_Thjf4UjNF9SAEwZjf8FZ0bw1iiXTIQ07qV5FhaLFilzUYYXzyAueNMLIfzb4MTKoY4EL-AR4hNKf1Gs1ac59QpdKWVG24mUTS5JmjX3cBi1CgXJxXVJjDWeFLcDNwvTHjY0v2U7wgX52irz1c8-Sk_y5VgF3dbjiKG9YENB8mXNBupXXMZy1Bg5dKuvjhy8hw6RASZeKIvLM28v7f76bTxD6KyyhTebE8Izu46XyWHZBfiN5qpCFTL3hHvv5PC3Du1hYpHZG8gR8RCtz-4WyYLZ4LQg8HKapelWEZRZKIgM4pp2gWoguUxmQ
Accept: */*
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
X-Requested-With: XMLHttpRequest
Content-Length: 311
Host: ac1udtlhapp08.epsilon.com
User-Agent: Apache-HttpClient/4.5.12 (Java/11.0.20.1)
</requestHeader>
<responseData class="java.lang.String">{"resultCode":"CLIENT_CONNECTION_ERROR","resultSubCode":"","serviceTransactionId":"393c6f53c32a4906a17a4ff449d14eae","clientRequestId":"393c6f53c32a4906a17a4ff449d14eae","data":["19de57f7-8bdd-4e9d-9e94-b2c9a20d05fd"],"total":0,"resultString":"The client either cannot connect to the server or received no response from the server."}</responseData>
<cookies class="java.lang.String"></cookies>
<method class="java.lang.String">PUT</method>
<queryString class="java.lang.String">{
"limitEmailDeliveryRate": false,
"name": "PERFHDSV2_PV4_MS250_25A25K_2M_Smartlink",
"id": "",
"type": "BATCH",
"deploymentDate":1701680645577, 
"audienceScheduleParams": {
"lockAudienceType": "SCHEDULE_TIME"
},
"contentType": "HTML",
"ignoreEmailChannelUnsubscribes":true
}
</queryString>
<java.net.URL>http://ac1udtlhapp08.epsilon.com/v1/messages/7a996de6-1b6e-4565-a3d7-d7c3893ce0b1/schedule</java.net.URL>
</httpSample>
Output of logstash using stdout -
{
"message" => "<httpSample t=\"45\" lt=\"45\" ts=\"1701467665726\" s=\"false\" lb=\"getTenantsSkynetCredentials\" rc=\"404\" rm=\"Not Found\" tn=\"TenantsAPis 1-3\" dt=\"text\" de=\"\" by=\"739\" ng=\"5\" na=\"5\">\n <responseHeader class=\"java.lang.String\">HTTP/1.1 404 Not Found\nDate: Fri, 01 Dec 2023 21:54:25 GMT\nContent-Type: application/json\nTransfer-Encoding: chunked\nConnection: keep-alive\nvary: Origin,Access-Control-Request-Method,Access-Control-Request-Headers\nx-client-request-id: daf2a05d-f8d8-405e-b830-a4360468d2bf\nx-content-type-options: nosniff\nx-xss-protection: 1; mode=block\ncache-control: no-cache, no-store, max-age=0, must-revalidate\npragma: no-cache\nexpires: 0\nx-frame-options: DENY\nx-envoy-upstream-service-time: 5\nserver: envoy\nset-cookie: SERVERID=nodeA; path=/\n</responseHeader>\n <requestHeader class=\"java.lang.String\">Connection: keep-alive\nAuthorization: Bearer eyJraWQiOiJGOV9JdThiV3QxT3labFhVekdhdWhpb0lpeVo2WUZLQ2Q4Z0ZFdTJySmt3IiwiYWxnIjoiUlMyNTYifQ.eyJ2ZXIiOjEsImp0aSI6IkFULjFrVWVYZ3hhVVp5bk05S2FmdGdCR3VsMWExZmR0UzN4MXdUSzhJNXdkYVkub2FyMmYzOGV3d2U0bjNKQmMwaDciLCJpc3MiOiJodHRwczovL3ByZXZpZXcucGVvcGxlY2xvdWQuZXBzaWxvbi5jb20vb2F1dGgyL2F1c3cwZnRoMW16SEJOdTdkMGg3IiwiYXVkIjoiZGF0YWh1YmFkbWluIiwiaWF0IjoxNzAxNDY2NTY1LCJleHAiOjE3MDE0NzAxNjUsImNpZCI6IkFESC1TVEdfU1ZOX09QRU5BTS1MRUdBQ1lfVEVTVF9URU5BTlRfU1ZDX0FDQ0VTUyIsInVpZCI6IjAwdXpiNTd4ZzVXY0RIMzN1MGg3Iiwic2NwIjpbIm9mZmxpbmVfYWNjZXNzIiwib3BlbmFtTGVnYWN5LmdpdmVubmFtZSIsIm9wZW5hbUxlZ2FjeS5lbXBsb3llZU51bWJlciIsIm9wZW5hbUxlZ2FjeS5jbiIsIm9wZW5hbUxlZ2FjeS5zbiIsIm9wZW5hbUxlZ2FjeS51aWQiLCJvcGVuYW1MZWdhY3kubWFpbCJdLCJhdXRoX3RpbWUiOjE3MDE0NjY1NjUsIm9wZW5hbUxlZ2FjeS5tYWlsIjoiSGFybW9ueV9FU19BcHBsaWNhdGlvbkBlcHNpbG9uLmNvbSIsInN1YiI6InRlc3RfdGVuYW50X2FjY2Vzc0BzdGcuZXBzaWxvbi5zZXJ2aWNlLmFjY291bnRzIiwib3BlbmFtTGVnYWN5LmNuIjoidGVzdF90ZW5hbnRfYWNjZXNzIHRlc3RfdGVuYW50X2FjY2VzcyIsIm9wZW5hbUxlZ2FjeS5zbiI6InRlc3RfdGVuYW50X2FjY2VzcyIsIm9wZW5hbUxlZ2FjeS5naXZlbm5hbWUiOiJ0ZXN0X3RlbmFudF9hY2Nlc3MiLCJvcGVuYW1MZWdhY3kudWlkIjoidGVzdF90ZW5hbnRfYWNjZXNzIn0.elmbizDOGucR3oZpjzPBVzdFtMxtIbJKEMZYBWzETf7mPcL44cgUc4a0zOwaJfu-rOw0Tbtv_p7wf-bKjJDYMr5KslPD-wZ45yEL8rbGa2gtwoEDtkPMyjMvoVifGJUA9wJ6v1TZow04kUU8EAbWAtOV2FdPqeVaaaGhgeugHSP3mNPvW-HnK61_6K_5geFkkUAaa95Fe2wqnNrLGzWlkYljTHNwT91vQWmqwADjUNsifbEkKiL9Mfk7BklUsC8elTDdaEDv924PgO8TbA02VIfvOZHZyrHdUc8Z0VbbpqT-x5fmtndI7fM6eMq1abcTYW4jf1ut_LB9HYBAhlmobw\nAccept: application/json\nHost: stgapi.datahub.epsilon.com\nUser-Agent: Apache-HttpClient/4.5.12 (Java/11.0.20.1)\n</requestHeader>\n <responseData class=\"java.lang.String\">{"errorList":[{"errorType":"CONSTRAINT","errorMessage":"UniteId and SkynetId is missing for tenant - beb55751-01a6-44ef-86f1-eb90b64f6268","errorProperty":null,"stacktrace":null}]}</responseData>\n <cookies class=\"java.lang.String\"></cookies>\n <method class=\"java.lang.String\">GET</method>\n <queryString class=\"java.lang.String\"></queryString>\n <java.net.URL>https://stgapi.datahub.epsilon.com/tenants-api/v1/tenants/beb55751-01a6-44ef-86f1-eb90b64f6268/skynetcredentials</java.net.URL>\n</httpSample>",
"tags" => [
[0] "multiline",
[1] "_dateparsefailure"
],
"request_url" => [
[0] "https://stgapi.datahub.epsilon.com/tenants-api/v1/tenants/beb55751-01a6-44ef-86f1-eb90b64f6268/skynetcredentials"
],
"label" => [
[0] "getTenantsSkynetCredentials"
],
"response_message" => [
[0] "Not Found"
],
"@timestamp" => 2023-12-05T23:44:17.780893300Z,
"request_header" => [
[0] "Connection: keep-alive\nAuthorization: Bearer eyJraWQiOiJGOV9JdThiV3QxT3labFhVekdhdWhpb0lpeVo2WUZLQ2Q4Z0ZFdTJySmt3IiwiYWxnIjoiUlMyNTYifQ.eyJ2ZXIiOjEsImp0aSI6IkFULjFrVWVYZ3hhVVp5bk05S2FmdGdCR3VsMWExZmR0UzN4MXdUSzhJNXdkYVkub2FyMmYzOGV3d2U0bjNKQmMwaDciLCJpc3MiOiJodHRwczovL3ByZXZpZXcucGVvcGxlY2xvdWQuZXBzaWxvbi5jb20vb2F1dGgyL2F1c3cwZnRoMW16SEJOdTdkMGg3IiwiYXVkIjoiZGF0YWh1YmFkbWluIiwiaWF0IjoxNzAxNDY2NTY1LCJleHAiOjE3MDE0NzAxNjUsImNpZCI6IkFESC1TVEdfU1ZOX09QRU5BTS1MRUdBQ1lfVEVTVF9URU5BTlRfU1ZDX0FDQ0VTUyIsInVpZCI6IjAwdXpiNTd4ZzVXY0RIMzN1MGg3Iiwic2NwIjpbIm9mZmxpbmVfYWNjZXNzIiwib3BlbmFtTGVnYWN5LmdpdmVubmFtZSIsIm9wZW5hbUxlZ2FjeS5lbXBsb3llZU51bWJlciIsIm9wZW5hbUxlZ2FjeS5jbiIsIm9wZW5hbUxlZ2FjeS5zbiIsIm9wZW5hbUxlZ2FjeS51aWQiLCJvcGVuYW1MZWdhY3kubWFpbCJdLCJhdXRoX3RpbWUiOjE3MDE0NjY1NjUsIm9wZW5hbUxlZ2FjeS5tYWlsIjoiSGFybW9ueV9FU19BcHBsaWNhdGlvbkBlcHNpbG9uLmNvbSIsInN1YiI6InRlc3RfdGVuYW50X2FjY2Vzc0BzdGcuZXBzaWxvbi5zZXJ2aWNlLmFjY291bnRzIiwib3BlbmFtTGVnYWN5LmNuIjoidGVzdF90ZW5hbnRfYWNjZXNzIHRlc3RfdGVuYW50X2FjY2VzcyIsIm9wZW5hbUxlZ2FjeS5zbiI6InRlc3RfdGVuYW50X2FjY2VzcyIsIm9wZW5hbUxlZ2FjeS5naXZlbm5hbWUiOiJ0ZXN0X3RlbmFudF9hY2Nlc3MiLCJvcGVuYW1MZWdhY3kudWlkIjoidGVzdF90ZW5hbnRfYWNjZXNzIn0.elmbizDOGucR3oZpjzPBVzdFtMxtIbJKEMZYBWzETf7mPcL44cgUc4a0zOwaJfu-rOw0Tbtv_p7wf-bKjJDYMr5KslPD-wZ45yEL8rbGa2gtwoEDtkPMyjMvoVifGJUA9wJ6v1TZow04kUU8EAbWAtOV2FdPqeVaaaGhgeugHSP3mNPvW-HnK61_6K_5geFkkUAaa95Fe2wqnNrLGzWlkYljTHNwT91vQWmqwADjUNsifbEkKiL9Mfk7BklUsC8elTDdaEDv924PgO8TbA02VIfvOZHZyrHdUc8Z0VbbpqT-x5fmtndI7fM6eMq1abcTYW4jf1ut_LB9HYBAhlmobw\nAccept: application/json\nHost: stgapi.datahub.epsilon.com\nUser-Agent: Apache-HttpClient/4.5.12 (Java/11.0.20.1)\n"
],
"response_data" => [
[0] "{\"errorList\":[{\"errorType\":\"CONSTRAINT\",\"errorMessage\":\"UniteId and SkynetId is missing for tenant - beb55751-01a6-44ef-86f1-eb90b64f6268\",\"errorProperty\":null,\"stacktrace\":null}]}"
],
"event" => {
"original" => "<httpSample t=\"45\" lt=\"45\" ts=\"1701467665726\" s=\"false\" lb=\"getTenantsSkynetCredentials\" rc=\"404\" rm=\"Not Found\" tn=\"TenantsAPis 1-3\" dt=\"text\" de=\"\" by=\"739\" ng=\"5\" na=\"5\">\n <responseHeader class=\"java.lang.String\">HTTP/1.1 404 Not Found\nDate: Fri, 01 Dec 2023 21:54:25 GMT\nContent-Type: application/json\nTransfer-Encoding: chunked\nConnection: keep-alive\nvary: Origin,Access-Control-Request-Method,Access-Control-Request-Headers\nx-client-request-id: daf2a05d-f8d8-405e-b830-a4360468d2bf\nx-content-type-options: nosniff\nx-xss-protection: 1; mode=block\ncache-control: no-cache, no-store, max-age=0, must-revalidate\npragma: no-cache\nexpires: 0\nx-frame-options: DENY\nx-envoy-upstream-service-time: 5\nserver: envoy\nset-cookie: SERVERID=nodeA; path=/\n</responseHeader>\n <requestHeader class=\"java.lang.String\">Connection: keep-alive\nAuthorization: Bearer eyJraWQiOiJGOV9JdThiV3QxT3labFhVekdhdWhpb0lpeVo2WUZLQ2Q4Z0ZFdTJySmt3IiwiYWxnIjoiUlMyNTYifQ.eyJ2ZXIiOjEsImp0aSI6IkFULjFrVWVYZ3hhVVp5bk05S2FmdGdCR3VsMWExZmR0UzN4MXdUSzhJNXdkYVkub2FyMmYzOGV3d2U0bjNKQmMwaDciLCJpc3MiOiJodHRwczovL3ByZXZpZXcucGVvcGxlY2xvdWQuZXBzaWxvbi5jb20vb2F1dGgyL2F1c3cwZnRoMW16SEJOdTdkMGg3IiwiYXVkIjoiZGF0YWh1YmFkbWluIiwiaWF0IjoxNzAxNDY2NTY1LCJleHAiOjE3MDE0NzAxNjUsImNpZCI6IkFESC1TVEdfU1ZOX09QRU5BTS1MRUdBQ1lfVEVTVF9URU5BTlRfU1ZDX0FDQ0VTUyIsInVpZCI6IjAwdXpiNTd4ZzVXY0RIMzN1MGg3Iiwic2NwIjpbIm9mZmxpbmVfYWNjZXNzIiwib3BlbmFtTGVnYWN5LmdpdmVubmFtZSIsIm9wZW5hbUxlZ2FjeS5lbXBsb3llZU51bWJlciIsIm9wZW5hbUxlZ2FjeS5jbiIsIm9wZW5hbUxlZ2FjeS5zbiIsIm9wZW5hbUxlZ2FjeS51aWQiLCJvcGVuYW1MZWdhY3kubWFpbCJdLCJhdXRoX3RpbWUiOjE3MDE0NjY1NjUsIm9wZW5hbUxlZ2FjeS5tYWlsIjoiSGFybW9ueV9FU19BcHBsaWNhdGlvbkBlcHNpbG9uLmNvbSIsInN1YiI6InRlc3RfdGVuYW50X2FjY2Vzc0BzdGcuZXBzaWxvbi5zZXJ2aWNlLmFjY291bnRzIiwib3BlbmFtTGVnYWN5LmNuIjoidGVzdF90ZW5hbnRfYWNjZXNzIHRlc3RfdGVuYW50X2FjY2VzcyIsIm9wZW5hbUxlZ2FjeS5zbiI6InRlc3RfdGVuYW50X2FjY2VzcyIsIm9wZW5hbUxlZ2FjeS5naXZlbm5hbWUiOiJ0ZXN0X3RlbmFudF9hY2Nlc3MiLCJvcGVuYW1MZWdhY3kudWlkIjoidGVzdF90ZW5hbnRfYWNjZXNzIn0.elmbizDOGucR3oZpjzPBVzdFtMxtIbJKEMZYBWzETf7mPcL44cgUc4a0zOwaJfu-rOw0Tbtv_p7wf-bKjJDYMr5KslPD-wZ45yEL8rbGa2gtwoEDtkPMyjMvoVifGJUA9wJ6v1TZow04kUU8EAbWAtOV2FdPqeVaaaGhgeugHSP3mNPvW-HnK61_6K_5geFkkUAaa95Fe2wqnNrLGzWlkYljTHNwT91vQWmqwADjUNsifbEkKiL9Mfk7BklUsC8elTDdaEDv924PgO8TbA02VIfvOZHZyrHdUc8Z0VbbpqT-x5fmtndI7fM6eMq1abcTYW4jf1ut_LB9HYBAhlmobw\nAccept: application/json\nHost: stgapi.datahub.epsilon.com\nUser-Agent: Apache-HttpClient/4.5.12 (Java/11.0.20.1)\n</requestHeader>\n <responseData class=\"java.lang.String\">{"errorList":[{"errorType":"CONSTRAINT","errorMessage":"UniteId and SkynetId is missing for tenant - beb55751-01a6-44ef-86f1-eb90b64f6268","errorProperty":null,"stacktrace":null}]}</responseData>\n <cookies class=\"java.lang.String\"></cookies>\n <method class=\"java.lang.String\">GET</method>\n <queryString class=\"java.lang.String\"></queryString>\n <java.net.URL>https://stgapi.datahub.epsilon.com/tenants-api/v1/tenants/beb55751-01a6-44ef-86f1-eb90b64f6268/skynetcredentials</java.net.URL>\n</httpSample>"
},
"method" => [
[0] "GET"
],
"response_header" => [
[0] "HTTP/1.1 404 Not Found\nDate: Fri, 01 Dec 2023 21:54:25 GMT\nContent-Type: application/json\nTransfer-Encoding: chunked\nConnection: keep-alive\nvary: Origin,Access-Control-Request-Method,Access-Control-Request-Headers\nx-client-request-id: daf2a05d-f8d8-405e-b830-a4360468d2bf\nx-content-type-options: nosniff\nx-xss-protection: 1; mode=block\ncache-control: no-cache, no-store, max-age=0, must-revalidate\npragma: no-cache\nexpires: 0\nx-frame-options: DENY\nx-envoy-upstream-service-time: 5\nserver: envoy\nset-cookie: SERVERID=nodeA; path=/\n"
],
"@version" => "1",
"time" => [
[0] 1701467665726
],
"host" => {
"name" => "187344ea2336"
},
"log" => {
"file" => {
"path" => "/input/error.jtl"
}
},
"type" => "xml",
"response_code" => [
[0] "404"
]
}
I am not seeing any parse failure ERROR in the stdout. The time captured is in epoch millis.