Hello everyone,
I'm trying to parse this two log lines :
2019-12-12T10:35:42.625+0100 ERROR log/input.go:501 Harvester could not be started on new file: /apps/tomcat/srvr_inf_tom80_dxccloud_dev_01/prod/logs/ju.log, Err: error setting up harvester: Harvester setup failed. Unexpected file opening error: Failed opening /apps/tomcat/srvr_inf_tom80_dxccloud_dev_01/prod/logs/ju.log: open /apps/tomcat/srvr_inf_tom80_dxccloud_dev_01/prod/logs/ju.log: permission denied
2019-12-12T11:44:16.298+0100 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":74680,"time":{"ms":1}},"total":{"ticks":1210750,"time":{"ms":9},"value":1210750},"user":{"ticks":1136070,"time":{"ms":8}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":13},"info":{"ephemeral_id":"3f78ff4f-9617-4183-a156-6252876d4dea","uptime":{"ms":360031499}},"memstats":{"gc_next":11677008,"memory_alloc":8535816,"memory_total":103981463128},"runtime":{"goroutines":70}},"filebeat":{"events":{"added":2,"done":2},"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"events":{"acked":2,"batches":2,"total":2}},"outputs":{"kafka":{"bytes_read":104,"bytes_write":2370}},"pipeline":{"clients":2,"events":{"active":0,"published":2,"total":2},"queue":{"acked":2}}},"registrar":{"states":{"current":2,"update":2},"writes":{"success":2,"total":2}},"system":{"load":{"1":0.08,"15":0.05,"5":0.03,"norm":{"1":0.08,"15":0.05,"5":0.03}}}}}}
My conf :
if [log][target][index] == "beats" {
dissect {
mapping => {
"message" => "%{timestamp} %{[log][level]} %{rest}"
}
add_tag => [ "parsed" ]
}
if [message] =~ /^.{0}[\[]/ {
dissect {
mapping => {
"rest" => "[%{[log][thread]}] %{[log][logger]} %{[log][message]}"
}
add_tag => [ "parsed" ]
}
} else {
dissect {
mapping => {
"rest" => "%{[log][logger]} %{[log][message]}"
}
add_tag => [ "parsed" ]
}
}
}
Result :
"rest": "[monitoring]\tlog/log.go:145\tNon-zero metrics in the last 30s\t{\"monitoring\": {\"metrics\": {\"beat\":{\"cpu\":{\"system\":{\"ticks\":1787406,\"time\":{\"ms\":16}},\"total\":{\"ticks\":2750484,\"time\":{\"ms\":16},\"value\":2750484},\"user\":{\"ticks\":963078}},\"handles\":{\"open\":302},\"info\":{\"ephemeral_id\":\"7100e63b-b5b6-4a79-b934-b9e40b9062bf\",\"uptime\":{\"ms\":2289181251}},\"memstats\":{\"gc_next\":8971424,\"memory_alloc\":6658576,\"memory_total\":100854298328},\"runtime\":{\"goroutines\":63}},\"filebeat\":{\"events\":{\"added\":1,\"done\":1},\"harvester\":{\"open_files\":1,\"running\":1}},\"libbeat\":{\"config\":{\"module\":{\"running\":0}},\"output\":{\"events\":{\"acked\":1,\"batches\":1,\"total\":1}},\"outputs\":{\"kafka\":{\"bytes_read\":52,\"bytes_write\":1116}},\"pipeline\":{\"clients\":1,\"events\":{\"active\":0,\"published\":1,\"total\":1},\"queue\":{\"acked\":1}}},\"registrar\":{\"states\":{\"current\":1,\"update\":1
"log": {
"offset": 6656040,
"file": {
"path": "C:\\Program Files\\FileBeats\\logs\\filebeat"
},
"target": {
"index": "beats",
"type": "filebeat"
},
"message": "log/log.go:145\tNon-zero metrics in the last 30s\t{\"monitoring\": {\"metrics\": {\"beat\":{\"cpu\":{\"system\":{\"ticks\":1787406,\"time\":{\"ms\":16}},\"total\":{\"ticks\":2750484,\"time\":{\"ms\":16},\"value\":2750484},\"user\":{\"ticks\":963078}},\"handles\":{\"open\":302},\"info\":{\"ephemeral_id\":\"7100e63b-b5b6-4a79-b934-b9e40b9062bf\",\"uptime\":{\"ms\":2289181251}},\"memstats\":{\"gc_next\":8971424,\"memory_alloc\":6658576,\"memory_total\":100854298328},\"runtime\":{\"goroutines\":63}},\"filebeat\":{\"events\":{\"added\":1,\"done\":1},\"harvester\":{\"open_files\":1,\"running\":1}},\"libbeat\":{\"config\":{\"module\":{\"running\":0}},\"output\":{\"events\":{\"acked\":1,\"batches\":1,\"total\":1}},\"outputs\":{\"kafka\":{\"bytes_read\":52,\"bytes_write\":1116}},\"pipeline\":{\"clients\":1,\"events\":{\"active\":0,\"published\":1,\"total\":1},\"queue\":{\"acked\":1}}},\"registrar\":{\"states\":{\"current\":1,\"update\":1},\"writes\":{\"success\":1,\"total\":1}}}}}",
"logger": "[monitoring]",
"level": "INFO"
}
The problem is that rest is not well parsed when I match "[", and [%{[log][thread]}] is ignored.
Thank you for helping