Hello Team,
grateful if you can help for the json parser. How can I modify the code below as per sample code?
input {
file {
start_position => "beginning"
path => "/data/KONG1/logs/b0197a09e1f3d7de67d47bdbc5f33f5b067a2ba224443704bf4d231cab913424/b0197a09e1f3d7de67d47bdbc5f33f5b067a2ba224443704bf4d231cab913424-json.log"
sincedb_path => "/dev/null"
}
}
filter {
json {
source => "message"
}
}
filter {
grok {
match => {"log" => "%{IP:clientip} - - [%{TIMESTAMP_ISO8601:timestamp}"}
}
}
output {
Elasticsearch {
hosts => "https://es1.innov.mt:9200"
index => "demo-json"
ssl => true
ssl_certificate_verification => false
cacert => "/etc/kibana/innov_mt.crt"
user => "mfslog"
password => "Yaxtubular6"
}
stdout {}
}
json sample file(1). To parse the whole json code (o.z.l.DefaultHttpLogWriter)
E [http-nio-8989-exec-2] o.z.l.DefaultHttpLogWriter: {"origin":"local","type":"response","correlation":"ffbbb6e7ea133c3f","duration":79,"protocol":"HTTP/1.1","status":200,"headers":{"Cache-Control":["no-cache, no-store, max-age=0, must-revalidate"],"Connection":["keep-alive"],"Content-Type":["application/json"],"Date":["Tue, 14 Jun 2022 14:31:23 GMT"],"Expires":["0"],"Keep-Alive":["timeout=20"],"Pragma":["no-cache"],"RCF":["/v1/bills/show-all-bill-acc/10000680/1"],"Set-Cookie":["JSESSIONID=E8A3610582818A6736D83E3B29CA1E01; Path=/mwallet; HttpOnly"],"Transfer-Encoding":["chunked"],"Vary":["Origin","Access-Control-Request-Method","Access-Control-Request-Headers","Origin","Access-Control-Request-Method","Access-Control-Request-Headers","Origin","Access-Control-Request-Method","Access-Control-Request-Headers"],"X-Content-Type-Options":["nosniff"],"X-Frame-Options":["DENY"],"X-TransctionId":["100366382108374"],"X-XSS-Protection":["1; mode=block"]},"body":"{\"billDetails\":[{\"billId\":\"E0FC6FEFBD046535E053B401F20AA6C4\",\"billerId\":\"4\",\"billerName\":\"CWA\",\"billerUserId\":\"111051\",\"lineNo\":\"DLB 7050\",\"accountNo\":\"17010061Q\",\"billName\":\"Home New\",\"totalAmount\":0.0,\"totalCount\":0,\"serviceId\":150,\"status\":\"Success\",\"error\":\"\",\"subBillDetails\":[]},{\"billId\":\"E0FC6FF077E76535E053B401F20AA6C4\",\"billerId\":\"2\",\"billerName\":\"my.t home\",\"billerUserId\":\"111050\",\"lineNo\":\"4185190\",\"accountNo\":\"5497265\",\"billName\":\"myhome\",\"totalAmount\":0.0,\"totalCount\":0,\"serviceId\":101,\"status\":\"Success\",\"error\":\"\",\"subBillDetails\":[]},{\"billId\":\"E0FC6FF094976535E053B401F20AA6C4\",\"billerId\":\"1\",\"billerName\":\"my.t postpay\",\"billerUserId\":\"111054\",\"lineNo\":\"52583224\",\"accountNo\":\"747249\",\"billName\":\"Dada Beeharry Mobile\",\"totalAmount\":0.0,\"totalCount\":0,\"serviceId\":25,\"status\":\"Success\",\"error\":\"\",\"subBillDetails\":[]},{\"billId\":\"E0FC6FF094996535E053B401F20AA6C4\",\"billerId\":\"2\",\"billerName\":\"my.t home\",\"billerUserId\":\"111050\",\"lineNo\":\"4642305\",\"accountNo\":\"5220292\",\"billName\":\"Dada fix\",\"totalAmount\":0.0,\"totalCount\":0,\"serviceId\":101,\"status\":\"Success\",\"error\":\"\",\"subBillDetails\":[]},{\"billId\":\"E0FC6FF12BA66535E053B401F20AA6C4\",\"billerId\":\"2\",\"billerName\":\"my.t home\",\"billerUserId\":\"111050\",\"lineNo\":\"4214828\",\"accountNo\":\"15424075\",\"billName\":\"Home New\",\"totalAmount\":0.0,\"totalCount\":0,\"serviceId\":101,\"status\":\"Success\",\"error\":\"\",\"subBillDetails\":[]},{\"billId\":\"E0FC6FEECFAA6535E053B401F20AA6C4\",\"billerId\":\"3\",\"billerName\":\"CEB\",\"billerUserId\":\"111052\",\"lineNo\":\"1..."}
json code no2:
[TRACE] 2022-06-15 16:08:41.168 (DefaultHttpLogWriter.java:79) -http-nio-8989-exec-10 - {"origin":"remote","type":"request","correlation":"a673b0c2e3ad2bac","protocol":"HTTP/1.1","remote":"10.242.1.98","method":"GET","uri":"http://10.242.1.110:8989/mwallet/v1/app-configurations","headers":{"accept-encoding":["gzip"],"apikey":["ME/eIRf113{,Es3+"],"authorization":["XXX"],"authtoken":["k+8L2WYQXYgbTgpk8EkduH+ml5ZYtKPvdD1JY9E72AnrBVFaScmrPxOkbQFPbzF/TwoOUvo76D9dzhHB3/3dp2n0kgW3py2PcLLJ1RwPSXbX82SwDmsVq1H1/cceEnxBl1NrJ8ztPmuCV0v2kKn4ekjWWCnDtHkjW9SXvBNEC8nEzhcvfDOqYw1r37Z6DOvDhzplfJ4TMb7H5ZtiIPsgJY9PqaPYyYE7816kK2shMgD4CHluF7x7BuEWM3cbecDDJTAr54CP/630VRu0+YFvPEpkrqWU+psrIZzEQ7LEVCGpCaGD6FFJhnMjJoHFjNFGwH+UjZIcG4sGha17ZxiZ95o6X6zL90ahy8XdEKCfdAoPykDExYt1sZV9k2miFSJuaG6SlaIlft+PS/mr7cyi/okg1jokqLZzwOyrQ685TSVHaM44d1eMAIGRLe4U7mtFOpEGNiWP4shPBJC6u5jFGo/c5MBO5MSiv8mxZlExhPThgnhUHkm5Cc1SreY+ZqGGv6JJtRil85pEQ+AbR3NSQZmxz0/R6tHIfdGZF7OPW1dnW1B14ZkVlrgFMy+7wr/uA9GAK3gjH9RKe4odMv/4ar9P9EydEz4XJ9sbE6Ew72KeUw8NsKXLCq5Ejn2cfMJMzNoD2slBOrX21n4S14as8aP0r4dDCePANwZWDTqkmZZTu5Lf+ORsLcR4WwjzVMwjmOUW0JLv9Y44tittTA/RTVHENU78LgVkgYenYY9ytZCnvTuKQg04tKuzKVftD4rg6zU6A99UqBwiQwlxPXA3SapHWLSAvE+WPQHxFKQqYRIokDC552mcbg3Yn9skuSUWVKqUG6k1W9QBuwf4QRLL6yRdSgwOw8BPXZrOvtZgYG4S5TRpTaqzM2Jfv0tTSfL5Ej7UuML7mzczyPIANXLjGcLHL/jFnBwUCj5zsJUbQifFUxBwkV2voY8asPBOVGux3I6WOCN77dL19CpQofaP0DT15Gp7Y/Lwtfu+5wsZo3yZ9bjYunfKtix1M39iVQ7d2eds+VmDeRkdT2s/kpMBmuOS3G0n22RsbCwP8nAkFC+zDS/I4pZk0MyLWn2thzH6bpMzsnz0XylCmzkrzWA15y2OohcNv9U5ijv8zvCqNRQ0DZbKZVXTz819qLMxR7KecSI0lNQyia6oSQ4utnd6ll0BANpy0WUcjSHmMNCSC3RYsMzDqFGYHfnDv2DRWCMhNVXIP4c+rFM2SgveTo5C5QBJQvqKH3C0Ug4YoO5KywKuRI59nHzCTH60yr2LldviKkf0CM95sHITnNkTyXzP+uiXjMIHRt1GU1iysvVNpBBli8fShT8J9c7wVZuVg0+0lkNCikwv4ooU2iSEMGkk7m2D95tuYOGNCZs2VNIw6WbB82NNKl1fQpOn3UICLl1xFRnPs+uTCouFOaGi9PiGogsI4qt+MzI/+tLvBspeEhk2TqcMfp12eKSRVu3Nv6Ypm2bytPtgeE71ICC/IoZU9RKsOEN0fdqb7IrrelICCz+BLHldMu67+orqLZWiIgbl/XTpMmo1fgtUJFs0H/iV1z3XIIIz9YP27EnQyaNGkiVHSqQDLzD/iXpNWK4BtYSD1CoLtOPzcIm3sIzeTU7LrKRDjMhhAVcuTyj1cmblLOVvQC+h7vwqqTfLHWUGw+KD3jEeNezRKCgbdeZ0EWwaDIqFHHZVHezc5Uzhi3tjOki8PD0jqr6Ld5D//h3rGXCLf2AEIwK4zFusSWbzAoOZYCIQICygmMAV7VXdPg3mlyCNsGceQNGRMA+1Prj2FWuvSTQ6nDgNLEIs0wZd+bf1qwjNbxTPUeBSbhR15bWxjOwLfrergb6QYsmHJ5AxScACr4XXJtPC6PWVvWVG8efRdh2WBU0y74z6YXqLA2U+scLSjXJssIHH+g=="],"channel":["ACA"],"connection":["keep-alive"],"content-type":["application/json"],"host":["10.242.1.110:8989"],"user-agent":["okhttp/4.9.3"],"x-auth-token":["eyJhbGciOiJIUzI1NiJ9.eyJBdXRob3JpdGllcyI6IjEyMDQ5LDEzMDE4LDEyMDI1LDEyMDQ0LDEwMjMsMTAwMDAsOTA1NSwxMzAxNiwxMzAxMSw5MDE3LDEwMDExLDkwMDQsMTYwMDMsOTA1NCw5MDMzLDEwMTQsMTIwNDYsMTAwMDEsMTAwMDMsMTAwNiw2MDAzLDEyMDIyLDEwMjYsNjAwMCwxMTAwMSwxMzAwMCwxMDI3LDEwMjEsMTMwMTAsMzAwMywxMDAwMiwxMDMzLDEzMDA1LDcwMDAsOTA0MiwxMzA0Miw5MDAyLDkwMzUsMTMwMTksOTA0NSwxMDE1LDEwMDE1LDkwMzcsMTAzNCwxMjAwMiw5MDM5LDMwMDIsMTIwMDcsMTIwNTMsMTYwMDYsMTIwNDcsNzAwMSwxMzA2MiwxMDE4LDEyMDU0LDkwMDcsMTIwNTEsMTcwMDUsMTAzMiwxMzA1OSwxMDE3LDcwMDIsMTMwMzAsOTAwMSwxNzAwMCwxNjAwMCwxMzAwNCwxMDAzLDEwMTEsNTAwMCwxMzAzMSwxMzAyOSwxMjA0MywxMzAxNCwyMDAyLDkwMDksMTAzMSwxMzAwNiwxNjAwMSw5MDA1LDcwMDMsMTcwMDEsOTA0MCwxNjAwNCwxMzAwMSw5MDQ5LDYwMDEsMTcwMDMsOTA0MywxMzAzOCw5MDQ0LDEwMDAsMTAwMSwxMDMwLDEwMTMsMTMwMzcsOTAyNiwxMjA0OCw5MDMxLDkwNTIsMTMwMjcsOTAzNCw5MDM4LDEwMDcsMTAwMDUsMTAyMiwxMzAyOCw2MDAyLDEwMTAsMTAwMTksOTAzNiwxMzA0MSw5MDEwLDEzMDI1LDkwMjgsOTAxMyw5MDQ2LDEyMDA2LDEyMDI0LDExMDAwLDEyMDQ1LDEwMTIsNTAwMiwxMzAzMiwxMzA2MywxMzA2NCwxMzAxNyw5MDQ4LDkwMDAsMTAwOCwxMzAyMiwxMzAzOSwxNzAwNCw5MDAzLDkwNTYsOTAyMSwxMDAxOCwyMDAxLDEzMDA3LDEyMDUyLDEzMDI2LDEzMDIxLDE2MDA1LDEwMjAsMTAwNCw5MDI1LDEzMDIwLDEyMDIzLDEyMDAwLDUwMDEsMTcwMDIsOTA0MSw5MDMyLDEwMDksMTAyNSwxMDI0LDkwNjksMTAwMDYsMTAxOSwxMDE2LDkwNjgsMjAwMCw5MDI3LDMwMDEsMTYwMDIsMTMwMTUsMzAwMCIsInN1YiI6IjEwMDA2MDgwIiwiaWF0IjoxNjU1MjkyMjczLCJleHAiOjE2NTc0NTIyNzN9.pZW4M7qtx1JYnIkqU3-jEsJBiXs8bIdG9qCj3YXPVYg"],"x-entity":["1"],"x-forwarded-for":["102.115.248.150"],"x-forwarded-host":["app.mytmoney.mu"],"x-forwarded-path":["/mwallet/v1/app-configurations"],"x-forwarded-port":["8083"],"x-forwarded-proto":["http"],"x-msisdn":["59189364"],"x-notificationservice":["fcm"],"x-real-ip":["102.115.248.150"],"x-userid":["10006080"]},"body":""}
Regards,
Roshan