I have all my desired fields coming into logstash under the message field, including the desired message. I want to pull them up one level. I am aware there are many similar topics and I have tried various techniques from them to no avail. 
This is my current parsed message as it is in Kibana:
"message": [
"{\"@timestamp\":\"2024-09-20T00:29:40.052Z\",
\"log.level\":\"info\",
\"message\":\"::ffff:172.22.0.1 - - [20/Sep/2024:00:29:40 +0000] \\\"POST /api/auth/sign-in?elevated=true HTTP/1.1\\\" 200 709 \\\"https://localhost:3000/\\\" \\\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36\\\"\",
\"http\":{\"version\":\"1.1\",
\"request\":{\"method\":\"POST\",\"headers\":{\"host\":\"localhost:86\",\"content-length\":\"344\",\"sec-ch-ua\":\"\\\"Chromium\\\";v=\\\"128\\\", \\\"Not;A=Brand\\\";v=\\\"24\\\", \\\"Google Chrome\\\";v=\\\"128\\\"\",\"content-type\":\"application/json\",\"sec-ch-ua-mobile\":\"?0\",\"user-agent\":\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36\",\"sec-ch-ua-platform\":\"\\\"Windows\\\"\",\"accept\":\"*/*\",\"origin\":\"https://localhost:3000\",\"sec-fetch-site\":\"same-site\",\"sec-fetch-mode\":\"cors\",\"sec-fetch-dest\":\"empty\",\"referer\":\"https://localhost:3000/\",\"accept-encoding\":\"gzip, deflate, br, zstd\",\"accept-language\":\"en-US,en;q=0.9\",\"cookie\":\"user-access-token=j%3Anull; security_authentication=Fe26.2**ff3ee23df7127b5e416a9a24af196ce63ce23f344bc3de1c59d90ed66314eeee*gA8ll9ldzsB1x-Q4E0dw-g*Pa4qKHqrXrRDZWiSY0M2Ahww-00kCEFMqz_NkLDS6RdTRzrmCU6C4KFgxX2I84Y0SGZO5jKGznEypV6G9hXAAhZHcPYjicGKt_Z3LazwO4CDL-HHU4uLZIzNx_a2g7qbAANb5VHEbjbF5OFOMhBnsdt-RtunPjLp73ZnB6Nb8hAtT3r2LNA9KenUO7U1fkkvf3a8q3zrs3q43CGJT9gvW8ux8xvtKNrLRpUW7RabDIkNL6noUcXIlpA1HIX1-Wy1**12b5c6082b8ccb9425080a9b17215cc62f9c8750bfbb87e90b6a917089fcfa3e*MvZiX5zkMtUheBMjsSmMHNfelRFCaZbw3FdIj5m9PGU\",\"priority\":\"u=1, i\"},\"body\":{\"bytes\":344}},\"response\":{\"status_code\":200,\"headers\":{\"x-powered-by\":\"Express\",\"access-control-allow-origin\":\"https://localhost:3000\",\"vary\":\"Origin\",\"access-control-allow-credentials\":\"true\",\"set-cookie\":[\"user-access-token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ0eXBlIjoiYWNjZXNzLXRva2VuIiwidXNlcm5hbWUiOiJoZWFsIiwicm9sZXMiOlt7InJvbGVfZ3VpZCI6IjA0NWQ1NTMxLTNkNmYtNGMyYS1hM2U5LWJkNWNhODc1YTAyNiIsImlzX2VsZXZhdGVkIjp0cnVlLCJwcm9qZWN0X2d1aWQiOm51bGwsInVzZXJuYW1lIjoiaGVhbCJ9LHsicm9sZV9ndWlkIjoiYjM3MDdlYzEtYTg2Yi00NWUyLWE4NjktOGM5OGE4MjhlMTViIiwiaXNfZWxldmF0ZWQiOmZhbHNlLCJwcm9qZWN0X2d1aWQiOm51bGwsInVzZXJuYW1lIjoiaGVhbCJ9LHsicm9sZV9ndWlkIjoiZTNjZjA4ZDctYTczYy00M2NhLThmNzItYzhlZmMxMmM2YmE4IiwiaXNfZWxldmF0ZWQiOnRydWUsInByb2plY3RfZ3VpZCI6bnVsbCwidXNlcm5hbWUiOiJoZWFsIn1dLCJpYXQiOjE3MjY3OTIxODAsImV4cCI6MTcyNjg3ODU4MCwiYXVkIjoiZGNtcyIsImlzcyI6ImRjbXMifQ.rBl8W0OKB2DiU844kVIbdBuC4x2vd31LHzls9AbLhJA; Path=/api/; Expires=Fri, 20 Sep 2024 01:29:40 GMT; HttpOnly; Secure\",\"user-refresh-token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ0eXBlIjoicmVmcmVzaC10b2tlbiIsInJvbGVzIjpbeyJyb2xlX2d1aWQiOiIwNDVkNTUzMS0zZDZmLTRjMmEtYTNlOS1iZDVjYTg3NWEwMjYiLCJpc19lbGV2YXRlZCI6dHJ1ZSwicHJvamVjdF9ndWlkIjpudWxsLCJ1c2VybmFtZSI6ImhlYWwifSx7InJvbGVfZ3VpZCI6ImIzNzA3ZWMxLWE4NmItNDVlMi1hODY5LThjOThhODI4ZTE1YiIsImlzX2VsZXZhdGVkIjpmYWxzZSwicHJvamVjdF9ndWlkIjpudWxsLCJ1c2VybmFtZSI6ImhlYWwifSx7InJvbGVfZ3VpZCI6ImUzY2YwOGQ3LWE3M2MtNDNjYS04ZjcyLWM4ZWZjMTJjNmJhOCIsImlzX2VsZXZhdGVkIjp0cnVlLCJwcm9qZWN0X2d1aWQiOm51bGwsInVzZXJuYW1lIjoiaGVhbCJ9XSwidXNlcm5hbWUiOiJoZWFsIiwiaWF0IjoxNzI2NzkyMTgwLCJleHAiOjE3MjY3OTU3ODAsImF1ZCI6ImRjbXMiLCJpc3MiOiJkY21zIn0.CZexGtPfp7FfPS9VfXE_cIa3IG6X3Mx4yZz51crmmZU; Path=/api/auth/refresh; Expires=Fri, 20 Sep 2024 01:29:40 GMT; HttpOnly; Secure\"],\"content-type\":\"application/json; charset=utf-8\",\"content-length\":\"709\",\"etag\":\"W/\\\"2c5-L+mHtuVJeUoAeD01UBF/wCxUgbk\\\"\"},\"body\":{\"bytes\":709}}},\"url\":{\"full\":\"https://localhost:86/api/auth/sign-in?elevated=true\",\"path\":\"/api/auth/sign-in\",\"query\":\"elevated=true\",\"domain\":\"localhost\"},\"client\":{\"address\":\"::ffff:172.22.0.1\",\"ip\":\"::ffff:172.22.0.1\",\"port\":34668},\"user_agent\":{\"original\":\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36\"}}"
],
This is the original message that arrived at Logstash:
{"level":"info",
"message":
["{\"@timestamp\":\"2024-09-20T00:29:40.052Z\",
\"log.level\":\"info\",
\"message\":\"::ffff:172.22.0.1 - - [20/Sep/2024:00:29:40 +0000] \\\"POST /api/auth/sign-in?elevated=true HTTP/1.1\\\" 200 709 \\\"https://localhost:3000/\\\" \\\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36\\\"\",
\"http\":{\"version\":\"1.1\",
\"request\":{\"method\":\"POST\",\"headers\":{\"host\":\"localhost:86\",\"content-length\":\"344\",\"sec-ch-ua\":\"\\\"Chromium\\\";v=\\\"128\\\", \\\"Not;A=Brand\\\";v=\\\"24\\\", \\\"Google Chrome\\\";v=\\\"128\\\"\",\"content-type\":\"application/json\",\"sec-ch-ua-mobile\":\"?0\",\"user-agent\":\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36\",\"sec-ch-ua-platform\":\"\\\"Windows\\\"\",\"accept\":\"*/*\",\"origin\":\"https://localhost:3000\",\"sec-fetch-site\":\"same-site\",\"sec-fetch-mode\":\"cors\",\"sec-fetch-dest\":\"empty\",\"referer\":\"https://localhost:3000/\",\"accept-encoding\":\"gzip, deflate, br, zstd\",\"accept-language\":\"en-US,en;q=0.9\",\"cookie\":\"user-access-token=j%3Anull; security_authentication=Fe26.2**ff3ee23df7127b5e416a9a24af196ce63ce23f344bc3de1c59d90ed66314eeee*gA8ll9ldzsB1x-Q4E0dw-g*Pa4qKHqrXrRDZWiSY0M2Ahww-00kCEFMqz_NkLDS6RdTRzrmCU6C4KFgxX2I84Y0SGZO5jKGznEypV6G9hXAAhZHcPYjicGKt_Z3LazwO4CDL-HHU4uLZIzNx_a2g7qbAANb5VHEbjbF5OFOMhBnsdt-RtunPjLp73ZnB6Nb8hAtT3r2LNA9KenUO7U1fkkvf3a8q3zrs3q43CGJT9gvW8ux8xvtKNrLRpUW7RabDIkNL6noUcXIlpA1HIX1-Wy1**12b5c6082b8ccb9425080a9b17215cc62f9c8750bfbb87e90b6a917089fcfa3e*MvZiX5zkMtUheBMjsSmMHNfelRFCaZbw3FdIj5m9PGU\",\"priority\":\"u=1, i\"},\"body\":{\"bytes\":344}},\"response\":{\"status_code\":200,\"headers\":{\"x-powered-by\":\"Express\",\"access-control-allow-origin\":\"https://localhost:3000\",\"vary\":\"Origin\",\"access-control-allow-credentials\":\"true\",\"set-cookie\":[\"user-access-token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ0eXBlIjoiYWNjZXNzLXRva2VuIiwidXNlcm5hbWUiOiJoZWFsIiwicm9sZXMiOlt7InJvbGVfZ3VpZCI6IjA0NWQ1NTMxLTNkNmYtNGMyYS1hM2U5LWJkNWNhODc1YTAyNiIsImlzX2VsZXZhdGVkIjp0cnVlLCJwcm9qZWN0X2d1aWQiOm51bGwsInVzZXJuYW1lIjoiaGVhbCJ9LHsicm9sZV9ndWlkIjoiYjM3MDdlYzEtYTg2Yi00NWUyLWE4NjktOGM5OGE4MjhlMTViIiwiaXNfZWxldmF0ZWQiOmZhbHNlLCJwcm9qZWN0X2d1aWQiOm51bGwsInVzZXJuYW1lIjoiaGVhbCJ9LHsicm9sZV9ndWlkIjoiZTNjZjA4ZDctYTczYy00M2NhLThmNzItYzhlZmMxMmM2YmE4IiwiaXNfZWxldmF0ZWQiOnRydWUsInByb2plY3RfZ3VpZCI6bnVsbCwidXNlcm5hbWUiOiJoZWFsIn1dLCJpYXQiOjE3MjY3OTIxODAsImV4cCI6MTcyNjg3ODU4MCwiYXVkIjoiZGNtcyIsImlzcyI6ImRjbXMifQ.rBl8W0OKB2DiU844kVIbdBuC4x2vd31LHzls9AbLhJA; Path=/api/; Expires=Fri, 20 Sep 2024 01:29:40 GMT; HttpOnly; Secure\",\"user-refresh-token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ0eXBlIjoicmVmcmVzaC10b2tlbiIsInJvbGVzIjpbeyJyb2xlX2d1aWQiOiIwNDVkNTUzMS0zZDZmLTRjMmEtYTNlOS1iZDVjYTg3NWEwMjYiLCJpc19lbGV2YXRlZCI6dHJ1ZSwicHJvamVjdF9ndWlkIjpudWxsLCJ1c2VybmFtZSI6ImhlYWwifSx7InJvbGVfZ3VpZCI6ImIzNzA3ZWMxLWE4NmItNDVlMi1hODY5LThjOThhODI4ZTE1YiIsImlzX2VsZXZhdGVkIjpmYWxzZSwicHJvamVjdF9ndWlkIjpudWxsLCJ1c2VybmFtZSI6ImhlYWwifSx7InJvbGVfZ3VpZCI6ImUzY2YwOGQ3LWE3M2MtNDNjYS04ZjcyLWM4ZWZjMTJjNmJhOCIsImlzX2VsZXZhdGVkIjp0cnVlLCJwcm9qZWN0X2d1aWQiOm51bGwsInVzZXJuYW1lIjoiaGVhbCJ9XSwidXNlcm5hbWUiOiJoZWFsIiwiaWF0IjoxNzI2NzkyMTgwLCJleHAiOjE3MjY3OTU3ODAsImF1ZCI6ImRjbXMiLCJpc3MiOiJkY21zIn0.CZexGtPfp7FfPS9VfXE_cIa3IG6X3Mx4yZz51crmmZU; Path=/api/auth/refresh; Expires=Fri, 20 Sep 2024 01:29:40 GMT; HttpOnly; Secure\"],\"content-type\":\"application/json; charset=utf-8\",\"content-length\":\"709\",\"etag\":\"W/\\\"2c5-L+mHtuVJeUoAeD01UBF/wCxUgbk\\\"\"},\"body\":{\"bytes\":709}}},\"url\":{\"full\":\"https://localhost:86/api/auth/sign-in?elevated=true\",\"path\":\"/api/auth/sign-in\",\"query\":\"elevated=true\",\"domain\":\"localhost\"},\"client\":{\"address\":\"::ffff:172.22.0.1\",\"ip\":\"::ffff:172.22.0.1\",\"port\":34668},\"user_agent\":{\"original\":\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36\"}}"]}
This is my current logstash configuration:
input {
file {
path => "/var/log/dcms/*.log"
start_position => "beginning"
codec => "json"
}
}
filter {
json {
source => "message"
remove_field => "message"
}
}
I basically want to pull out all the fields inside message into filterable fields in Kibana.
As I understand, my filter should be reading the message field and parsing it as JSON, and putting them in the top-level of the message, then removing the original message field. But it seems to have no effect on my logs.