How to Filter log base on log line

[Bhargav_Patel]

Hi I am newbee in logstash.

can someone let me know how to filter below log. I want ot break this line base on timestamp java class and my custom text like "Updated By" and "Audit Log" and log level etc.

2018-04-05 17:22:02,047 DEBUG [http-nio-8082-exec-1] DomainResource: Audit Log : Updating user information. User Id is AAA11BE | Updated By : bhargav

I have tried below filter but not working.

            input {

file {
	path => ["/home/truecom/PE/portaleditor.log"]
	type => "pelog"
	start_position => end
	ignore_older => 0
	sincedb_path => "null"
}

}filter {

mutate {
gsub => ['message', "\n", " "]
}
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:time} %{LOGLEVEL:loglevel} %{JAVACLASS:class} - %{GREEDYDATA:msg}" }
}
}output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => ["localhost:9200"]
action => "index"
index => "portalditor-%{+YYYY.MM.dd}"
}
}

Kibana output is

path:/home/truecom/PE/portaleditor.log @timestamp:April 5th 2018, 18:54:37.678 @version:1 host:truecom message:2018-04-05 18:54:36,690 DEBUG [http-nio-8082-exec-9] ClientResource: Audit Log : Getting allClients type:pelog tags:_grokparsefailure _id:AWKV-wDXbTkgN2yFEvrM _type:pelog _index:portalditor-2018.04.05 _score:

Logstash Output

          "path" => "/home/truecom/PE/portaleditor.log",
"@timestamp" => 2018-04-05T13:32:21.854Z,
  "@version" => "1",
      "host" => "truecom",
   "message" => "2018-04-05 19:02:21,499 DEBUG [http-nio-8082-exec-5] ClientResource: Audit Log : Getting allClients",
      "type" => "pelog",
      "tags" => [
    [0] "_grokparsefailure"
]

I want to extract message field. Like log level and timestamp and msg.

[Bhargav_Patel]

Some how i am able to get the data like log level but i am getting blank for GREEDYDATA. what is wrong in my config file? i tried to put GREEDYDATA at the end as well. belows my update .conf file .

input {

file {
	path => ["/home/truecom/PE/portaleditor.log"]
	type => "pelog"
	start_position => end
	ignore_older => 0
	sincedb_path => "null"
}

}
filter {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}%{GREEDYDATA:input}%{LOGLEVEL:loglevel}" }
}

}
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => ["localhost:9200"]
action => "index"
index => "portalditor-%{+YYYY.MM.dd}"
}
}

Log stash output is

{
      "path" => "/home/truecom/PE/portaleditor.log",
     "input" => " ",
"@timestamp" => 2018-04-05T14:01:24.784Z,
  "loglevel" => "DEBUG",
  "@version" => "1",
      "host" => "truecom",
   "message" => "2018-04-05 19:31:24,293 DEBUG [http-nio-8082-exec-2] ClientResource: Audit Log : Getting allClients",
      "type" => "pelog",
 "timestamp" => "2018-04-05 19:31:24,293"

}

[Bhargav_Patel]

Getting more closer now. able to extract timestep and log level. but facing issue while getting thead and java class

grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}%{THREAD:thread}%{SPACE}%{GREEDYDATA:msg}" }

}

Getting _grokparsefailure while adding THREAD and JAVACLASS

Logstash error is

error=>"pattern %{THREAD:thread} not defined"}

[magnusbaeck (Magnus Bäck)]

I don't believe THREAD is a standard grok pattern. What gave you that idea that it was? You can e.g. use

\[(?<thread>[^\]]*)\]

to match the thread name (capture everything between the opening square bracket and the first closing one).

[Bhargav_Patel]

Thanks for quick reply.

Not sure about the solution you have mention but this %{NOTSPACE:javathread} work for me. will also try the same which you have mention.

[magnusbaeck (Magnus Bäck)]

NOTSPACE works too but will capture the square brackets that aren't really part of the thread name. You could of course remove the square brackets with a mutate filter.