Sorry, I had to scrape all the confidential information before posting here. But this is what I see in the logstash log once I enabled the stdout output section:
[2022-02-23T14:05:48,918][INFO ][logstash.javapipeline ][filebeat-logstash1] Pipeline started {"pipeline.id"=>"filebeat-logstash1"}
[2022-02-23T14:05:49,056][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:"filebeat-logstash1"], :non_running_pipelines=>[]}
[2022-02-23T14:05:49,059][INFO ][filewatch.observingtail ][filebeat-logstash1] START, creating Discoverer, Watch with file and sincedb collections
[2022-02-23T14:05:49,997][WARN ][logstash.filters.json ][filebeat-logstash1] Error parsing json {:source=>"message", :raw=>"{\"id\":\"1f9f9f9f9-ba63-41fd-9665-b8484086cb5c\",\"message\":\"Claimed 1 authorization out of 1 unclaimed matches, 0 matches were already claimed.\",\"claimedAuthorization\":{\"id\":\"1f9f9f9f9-ba63-41fd-9665-b8484086cb5c\",\"token\":\"5111114QXZEFC1222\",\"expirationMMYY\":\"1111\",\"referenceDate\":\"2022-01-06\",\"amount\":1031.22,\"currencyCode\":\"USD\",\"nameOnCard\":\"User T Test\",\"telephoneNumber\":\"1111111111\",\"address1\":\"PO BOX 1111\",\"city\":\"CITY\",\"stateOrProvinceCode\":\"CA\",\"postalCode\":\"11111\",\"countryCode\":\"US\",\"channel\":\"WEB\",\"authorizationProperties\":{\"orderNumber\":\"64556572\",\"webCartId\":\"\",\"webSessionId\":\"cdfaf57a-853b-431e-a2bf-2d576c4e9980\",\"webIP\":\"111.111.11.11\",\"invoiceNumber\":\"111111111\",\"customerNumber\":\"1111111\"},\"verified\":true,\"authorized\":true,\"authorizationResponseCode\":\"100\",\"authorizationResponseDescription\":\"APPROVED\",\"authorizationCode\":\"038394\",\"addressVerificationResponseCode\":\"I4\",\"level3\":\"N\",\"commercialCreditCard\":\"Y\",\"overrideModeOfPayment\":\"MC\",\"transactionId\":null,\"cvsResponse\":null,\"cvvMatchResponseCode\":null,\"createdTimestamp\":\"2022-02-10T10:55:53.115-06:00[America/Chicago\\\",\"claimed\":true,\"claimedTimestamp\":\"2022-02-10T10:56:29.198-06:00[America/Chicago\\\",\"claimedRequestId\":\"33a98688-d8ae-4c5f-94d9-986393bd1406\",\"merchantOrderNumber\":\"64556572\"}}]", :exception=>#<LogStash::Json::ParserError: Unexpected character ('c' (code 99)): was expecting comma to separate Object entries
at [Source: (byte[])"{"id":"1f9f9f9f9-ba63-41fd-9665-b8484086cb5c","message":"Claimed 1 authorization out of 1 unclaimed matches, 0 matches were already claimed.","claimedAuthorization":{"id":"1f9f9f9f9-ba63-41fd-9665-b8484086cb5c","token":"5111114QXZEFC1222","expirationMMYY":"1111","referenceDate":"2022-01-06","amount":1031.22,"currencyCode":"USD","nameOnCard":"User T Test","telephoneNumber":"1111111111","address1":"PO BOX 1111","city":"CITY","stateOrProvinceCode":"CA","postalCode":"11111","countryCode":"US","[truncated 780 bytes]; line: 1, column: 1107]>}
{
"restType" => "RESPONSE",
"userName" => "APIUSER",
"responseCode" => 200,
"loggerName" => "com.comp.ha.rest.util.RestLoggingHelper",
"tags" => [
[0] "file-based",
[1] "_jsonparsefailure"
],
"@version" => "1",
"logFilePath" => "%{[log][file][path]}",
"logLevel" => "info",
"port" => 10610,
"correlation-id" => "A4844003AB991A7AA9AB0004AC1C0C7C",
"zone" => "backoffice",
"@timestamp" => 2022-02-23T19:56:29.205Z,
"message" => "RESPONSE BODY : %{parsedccclaimrespjson}",
"hostname" => "%{[host][name]}",
"request-id" => "9Q8V1n5WRmmt-Xpj4EmlXw",
"originatorApp" => "test-api",
"userId" => "",
"jvm" => "test-api-blue04-dc1-zn1",
"threadName" => "Default Executor-thread-2453",
"path" => "/pathfilebeat/config/test.log",
"appApp" => "test-api[v1]",
"responseStatus" => "OK",
"type" => "log",
"originatorContext" => "/credit-card/authorization-claims"
}
[2022-02-23T14:05:51,576][DEBUG][logstash.outputs.elasticsearch][filebeat-logstash1] Sending final bulk request for batch. {:action_count=>1, :payload_size=>835, :content_length=>835, :batch_offset=>0}
When I sent the working message with all test data in it, here is the output:
[2022-02-23T14:18:34,182][WARN ][logstash.filters.json ][logstash-filebeat1] Parsed JSON object/hash requires a target configuration option {:source=>"message", :raw=>""}
{
"@version" => "1",
"hostname" => "%{[host][name]}",
"type" => "log",
"tags" => [
[0] "_jsonparsefailure",
[1] "file-based"
],
"message" => "",
"zone" => "backoffice",
"logFilePath" => "%{[log][file][path]}",
"path" => "path/logstashpipline/config/test2.log",
"@timestamp" => 2022-02-23T20:18:34.038Z
}
{
"originatorApp" => "test-api",
"originatorContext" => "/credit-card/authorization-claims",
"userName" => "test",
"loggerName" => "com.ha.rest.RestLoggingHelper",
"logLevel" => "info",
"@version" => "1",
"userId" => "",
"hostname" => "%{[host][name]}",
"type" => "log",
"message" => "restType=RESPONSE|requestUri=/test-api/cc/auth|responseCode=200|responseStatus=OK|requestBody=[{\"channel\":\"CCC\",\"criteria\":[{\"token\":\"XXXXXXXXXX\",\"referenceDate\":\"2022-01-06\",\"amount\":0000.22,\"authorizationProperties\":{\"customerNumber\":\"1111111\",\"orderNumber\":\"211212121\"}}]} \u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000]|responseBody=[{\"id\":\"121212121-12121-12121-b8484086cb5c\",\"message\":\"Claimed 1 authorization out of 1 unclaimed matches, 0 matches were already claimed.\",\"claimedAuthorization\":{\"id\":\"121212121-12121-12121-b8484086cb5c\",\"token\":\"121212121121212\",\"expirationMMYY\":\"0000\",\"referenceDate\":\"2022-01-06\",\"amount\":0000.22,\"currencyCode\":\"USD\",\"nameOnCard\":\"test t. user\",\"telephoneNumber\":\"121212121212\",\"address1\":\"PO BOX 1110000\",\"city\":\"test\",\"stateOrProvinceCode\":\"CA\",\"postalCode\":\"00000\",\"countryCode\":\"US\",\"channel\":\"CCC\",\"authorizationProperties\":{\"orderNumber\":\"1212121212\",\"webCartId\":\"\",\"webSessionId\":\"cdfaf57a-853b-431e-a2bf-12121212121\",\"webIP\":\"000.000.10.00\",\"invoiceNumber\":\"12121212\",\"customerNumber\":\"12121212\"},\"verified\":true,\"authorized\":true,\"authorizationResponseCode\":\"100\",\"authorizationResponseDescription\":\"APPROVED\",\"authorizationCode\":\"1212121\",\"addressVerificationResponseCode\":\"I4\",\"level3\":\"N\",\"commercialCreditCard\":\"Y\",\"overrideModeOfPayment\":\"MC\",\"transactionId\":null,\"cvsResponse\":null,\"cvvMatchResponseCode\":null,\"createdTimestamp\":\"2022-02-13T10:55:53.115-06:00[America/Chicago]\",\"claimed\":true,\"claimedTimestamp\":\"2022-02-13T10:56:29.198-06:00[America/Chicago]\",\"claimedRequestId\":\"121212121-d8ae-4c5f-94d9-986393bd1406\",\"merchantOrderNumber\":\"12121212121\"}}]",
"logFilePath" => "%{[log][file][path]}",
"threadName" => "Default Executor-thread-2453",
"port" => 10610,
"jvm" => "test-api-blue04",
"app" => "test-api[v1]",
"tags" => [
[0] "file-based"
],
"zone" => "backoffice",
"request-id" => "9Q8V1n5WRmmt-Xpj4EmlXw",
"path" => "path/logstashpipline/config/test2.log",
"@timestamp" => 2022-02-23T19:56:29.205Z,
"correlation-id" => "121212121212121"
}
