Logstash filter from Logstash-forwarder

Elastic Stack Logstash
1

I am trying to get my logs work right however cannot work out what I am going wrong.

filter{
	grok{
		match => [ "message", "%{TIMESTAMP_ISO8601:timestamp}% %{LOGLEVEL:loglevel}% %{GREEDYDATA:message}" ]
		overwrite => ["message"]
	}
	json{
		source => "message"
		remove_field => ["message"]
	}
}

My logs are

[2015-08-20 14:00:48.195524] [info] {"JOBID":"","USERID":"","TYPE":"","REF":"","INFO":"ps x | grep .php | awk '{print $7}'"}
[2015-08-20 14:00:48.209359] [info] {"JOBID":"","USERID":"","TYPE":"","REF":"","INFO":"Watching tube: guv"}
[2015-08-20 14:01:37.489907] [info] {"JOBID":"","USERID":"","TYPE":"","REF":"","INFO":"JobsProcessing: New tube announced (S.90.1)"}
[2015-08-20 14:01:37.493200] [info] {"JOBID":"","USERID":"","TYPE":"","REF":"","INFO":"S.90.1 TRUE  Is_Worker_Needed_For_This_Tube"}
[2015-08-20 14:01:37.493978] [info] {"JOBID":"","USERID":"","TYPE":"","REF":"","INFO":"S.90.1 Match on  Get_Worker_File"}
[2015-08-20 14:01:37.496040] [info] {"JOBID":"","USERID":"","TYPE":"","REF":"","INFO":"S.90.1 Match on  Get_Worker_File"}
[2015-08-20 14:01:37.496711] [info] {"JOBID":"","USERID":"","TYPE":"","REF":"","INFO":"ps x | grep \"[m]ock_worker.php S.90.1\" | awk '{print $7}'"}
[2015-08-20 14:01:37.497358] [info] {"JOBID":"","USERID":"","TYPE":"","REF":"","INFO":"S.90.1 Match on  Get_Worker_File"}
[2015-08-20 14:01:37.509294] [info] {"JOBID":"","USERID":"","TYPE":"","REF":"","INFO":"MakeProcess:(S.90.1)"}

If anyone has the time to give me some advice that would be great thank you, also note I can change the log files to what ever I thought it was easier to use JSON as this is using php klogger.

2

There are a couple of problems here:

  • It's %{PATTERN:fieldname}, not %{PATTERN:fieldname}%.
  • The timestamp and log level are surrounded by square brackets but they're not included in your grok expression: Hence: \[%{TIMESTAMP_ISO8601:timestamp}\] \[%{LOGLEVEL:loglevel}\].
1 Like
3

Your a super star thank you!