Dissect not parsing

Elastic Stack Logstash
1

Hello everyone,

I'm trying to parse this two log lines :

2019-12-12T10:35:42.625+0100	ERROR	log/input.go:501	Harvester could not be started on new file: /apps/tomcat/srvr_inf_tom80_dxccloud_dev_01/prod/logs/ju.log, Err: error setting up harvester: Harvester setup failed. Unexpected file opening error: Failed opening /apps/tomcat/srvr_inf_tom80_dxccloud_dev_01/prod/logs/ju.log: open /apps/tomcat/srvr_inf_tom80_dxccloud_dev_01/prod/logs/ju.log: permission denied
2019-12-12T11:44:16.298+0100	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":74680,"time":{"ms":1}},"total":{"ticks":1210750,"time":{"ms":9},"value":1210750},"user":{"ticks":1136070,"time":{"ms":8}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":13},"info":{"ephemeral_id":"3f78ff4f-9617-4183-a156-6252876d4dea","uptime":{"ms":360031499}},"memstats":{"gc_next":11677008,"memory_alloc":8535816,"memory_total":103981463128},"runtime":{"goroutines":70}},"filebeat":{"events":{"added":2,"done":2},"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"events":{"acked":2,"batches":2,"total":2}},"outputs":{"kafka":{"bytes_read":104,"bytes_write":2370}},"pipeline":{"clients":2,"events":{"active":0,"published":2,"total":2},"queue":{"acked":2}}},"registrar":{"states":{"current":2,"update":2},"writes":{"success":2,"total":2}},"system":{"load":{"1":0.08,"15":0.05,"5":0.03,"norm":{"1":0.08,"15":0.05,"5":0.03}}}}}}

My conf :

if [log][target][index] == "beats" {
  
		dissect {
        mapping => {
        "message" => "%{timestamp}	%{[log][level]}	%{rest}"
            }
				add_tag => [ "parsed" ]
    }
	  if [message] =~ /^.{0}[\[]/ {
		dissect {
        mapping => {
        "rest" => "[%{[log][thread]}]	%{[log][logger]}    %{[log][message]}"
            }
				add_tag => [ "parsed" ]
    }
	  
	  } else {
		dissect {
        mapping => {
        "rest" => "%{[log][logger]}	%{[log][message]}"
            }
				add_tag => [ "parsed" ]
    }
	  }
}

Result :

"rest": "[monitoring]\tlog/log.go:145\tNon-zero metrics in the last 30s\t{\"monitoring\": {\"metrics\": {\"beat\":{\"cpu\":{\"system\":{\"ticks\":1787406,\"time\":{\"ms\":16}},\"total\":{\"ticks\":2750484,\"time\":{\"ms\":16},\"value\":2750484},\"user\":{\"ticks\":963078}},\"handles\":{\"open\":302},\"info\":{\"ephemeral_id\":\"7100e63b-b5b6-4a79-b934-b9e40b9062bf\",\"uptime\":{\"ms\":2289181251}},\"memstats\":{\"gc_next\":8971424,\"memory_alloc\":6658576,\"memory_total\":100854298328},\"runtime\":{\"goroutines\":63}},\"filebeat\":{\"events\":{\"added\":1,\"done\":1},\"harvester\":{\"open_files\":1,\"running\":1}},\"libbeat\":{\"config\":{\"module\":{\"running\":0}},\"output\":{\"events\":{\"acked\":1,\"batches\":1,\"total\":1}},\"outputs\":{\"kafka\":{\"bytes_read\":52,\"bytes_write\":1116}},\"pipeline\":{\"clients\":1,\"events\":{\"active\":0,\"published\":1,\"total\":1},\"queue\":{\"acked\":1}}},\"registrar\":{\"states\":{\"current\":1,\"update\":1
"log": {
  "offset": 6656040,
  "file": {
    "path": "C:\\Program Files\\FileBeats\\logs\\filebeat"
  },
  "target": {
    "index": "beats",
    "type": "filebeat"
  },
  "message": "log/log.go:145\tNon-zero metrics in the last 30s\t{\"monitoring\": {\"metrics\": {\"beat\":{\"cpu\":{\"system\":{\"ticks\":1787406,\"time\":{\"ms\":16}},\"total\":{\"ticks\":2750484,\"time\":{\"ms\":16},\"value\":2750484},\"user\":{\"ticks\":963078}},\"handles\":{\"open\":302},\"info\":{\"ephemeral_id\":\"7100e63b-b5b6-4a79-b934-b9e40b9062bf\",\"uptime\":{\"ms\":2289181251}},\"memstats\":{\"gc_next\":8971424,\"memory_alloc\":6658576,\"memory_total\":100854298328},\"runtime\":{\"goroutines\":63}},\"filebeat\":{\"events\":{\"added\":1,\"done\":1},\"harvester\":{\"open_files\":1,\"running\":1}},\"libbeat\":{\"config\":{\"module\":{\"running\":0}},\"output\":{\"events\":{\"acked\":1,\"batches\":1,\"total\":1}},\"outputs\":{\"kafka\":{\"bytes_read\":52,\"bytes_write\":1116}},\"pipeline\":{\"clients\":1,\"events\":{\"active\":0,\"published\":1,\"total\":1},\"queue\":{\"acked\":1}}},\"registrar\":{\"states\":{\"current\":1,\"update\":1},\"writes\":{\"success\":1,\"total\":1}}}}}",
  "logger": "[monitoring]",
  "level": "INFO"
}

The problem is that rest is not well parsed when I match "[", and [%{[log][thread]}] is ignored.

Thank you for helping

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.