Logstash 5.2.0 - _grokparsefailure, but it looks like the grok filter is correctly working

Elastic Stack Logstash
1

If I tail -f /var/log/messages, I see:

...
{
Mar  6 12:18:16 s-ut-logstash-1 logstash: "server" => "443",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "referer" => "\"https://broadcast.storagecraft.com/videos/\"",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "auth" => "-",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "ident" => "-",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "bytes_received" => "993",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "source" => "/var/log/httpd/broadcast.storagecraft.com_log",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "request_method" => "GET",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "type" => "log",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "uri_path" => "/videos/livestreams/json/",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "@version" => "1",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "beat" => {
Mar  6 12:18:16 s-ut-logstash-1 logstash: "hostname" => "redacted",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "name" => "redacted",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "version" => "5.2.0"
Mar  6 12:18:16 s-ut-logstash-1 logstash: },
Mar  6 12:18:16 s-ut-logstash-1 logstash: "host" => "redacted",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "client_ip" => "redacted",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "user_agent" => "\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36\"",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "geoip" => {},
Mar  6 12:18:16 s-ut-logstash-1 logstash: "offset" => 75032,
Mar  6 12:18:16 s-ut-logstash-1 logstash: "time_stamp" => "06/Mar/2017:12:18:01 -0700",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "input_type" => "log",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "http_version" => "1.1",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "message" => "redacted - - [06/Mar/2017:12:18:01 -0700] \"broadcast.storagecraft.com\" 443 \"GET /videos/livestreams/json/ HTTP/1.1\" 200 \"https://broadcast.storagecraft.com/videos/\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36\" 993 511",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "bytes_sent" => "511",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "tags" => [
Mar  6 12:18:16 s-ut-logstash-1 logstash: [0] "beats_input_codec_plain_applied",
Mar  6 12:18:16 s-ut-logstash-1 logstash: [1] "_grokparsefailure",     <======================================
Mar  6 12:18:16 s-ut-logstash-1 logstash: [2] "_geoip_lookup_failure"
Mar  6 12:18:16 s-ut-logstash-1 logstash: ],
Mar  6 12:18:16 s-ut-logstash-1 logstash: "@timestamp" => 2017-03-06T19:18:11.750Z,
Mar  6 12:18:16 s-ut-logstash-1 logstash: "response" => "200",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "web_site" => "\"broadcast.storagecraft.com\""
Mar  6 12:18:16 s-ut-logstash-1 logstash: }
...

But it looks like the filter is correctly tagging all the fields I wanted to create. I tested with the Grok Debugger, and it all looks good:

input:

14.96.134.214 - - [06/Mar/2017:10:37:29 -0700] 443 "www.storagecraft.com" "GET /sites/default/files/css/css_GfNox0nNkNOiad9gOPpDq4UyJGp-V37aPhgl0agiTT8.css?omel7z HTTP/1.1" 200 39339 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 1502 40272

pattern:

%{IP:client_ip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:time_stamp}\] %{NUMBER:server_port} %{QS:web_site} \"%{WORD:request_method} %{URIPATHPARAM:uri_path} HTTP/%{NUMBER:http_version}\" %{NUMBER:response} (?:%{NUMBER:bytes_transfered}|-) (?:%{QS:referer}|-) %{QS:user_agent} %{NUMBER:bytes_received} %{NUMBER:bytes_sent}

Should I be concerned, or just leave it?

2

I think you should get to the bottom of what's happening. Do you have additional files in /etc/logstash/conf.d besides the one(s) you expect Logstash to use? Remember that it reads all files in that directory.

3

I did not know that. I did make a copy of my .conf file as a backup (called it filename.conf.bak) and kept it in that directory. I'll remove it and test.

4

It appears that was the issue, and I will of course mark your reply as the answer, but I wonder if you could answer a follow up for me. Making that change no longer sends all beat logs to /var/log/messages. Instead I get this:

Mar  6 09:39:13 s-ut-logstash-1 logstash: Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties

However, those logs do not appear anywhere in /var/log/logstash. The logs I am referring to are these:

{
Mar  6 12:18:16 s-ut-logstash-1 logstash: "server" => "443",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "referer" => "\"https://broadcast.storagecraft.com/videos/\"",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "auth" => "-",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "ident" => "-",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "bytes_received" => "993",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "source" => "/var/log/httpd/broadcast.storagecraft.com_log",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "request_method" => "GET",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "type" => "log",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "uri_path" => "/videos/livestreams/json/",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "@version" => "1",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "beat" => {
Mar  6 12:18:16 s-ut-logstash-1 logstash: "hostname" => "redacted",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "name" => "redacted",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "version" => "5.2.0"
Mar  6 12:18:16 s-ut-logstash-1 logstash: },
Mar  6 12:18:16 s-ut-logstash-1 logstash: "host" => "redacted",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "client_ip" => "redacted",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "user_agent" => "\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36\"",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "geoip" => {},
Mar  6 12:18:16 s-ut-logstash-1 logstash: "offset" => 75032,
Mar  6 12:18:16 s-ut-logstash-1 logstash: "time_stamp" => "06/Mar/2017:12:18:01 -0700",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "input_type" => "log",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "http_version" => "1.1",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "message" => "redacted - - [06/Mar/2017:12:18:01 -0700] \"broadcast.storagecraft.com\" 443 \"GET /videos/livestreams/json/ HTTP/1.1\" 200 \"https://broadcast.storagecraft.com/videos/\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36\" 993 511",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "bytes_sent" => "511",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "tags" => [
Mar  6 12:18:16 s-ut-logstash-1 logstash: [0] "beats_input_codec_plain_applied",
Mar  6 12:18:16 s-ut-logstash-1 logstash: [1] "_grokparsefailure",     <======================================
Mar  6 12:18:16 s-ut-logstash-1 logstash: [2] "_geoip_lookup_failure"
Mar  6 12:18:16 s-ut-logstash-1 logstash: ],
Mar  6 12:18:16 s-ut-logstash-1 logstash: "@timestamp" => 2017-03-06T19:18:11.750Z,
Mar  6 12:18:16 s-ut-logstash-1 logstash: "response" => "200",
Mar  6 12:18:16 s-ut-logstash-1 logstash: "web_site" => "\"broadcast.storagecraft.com\""
Mar  6 12:18:16 s-ut-logstash-1 logstash: }
5

It looks like your .bak file had a stdout { codec => rubydebug } output that you wanted to keep around.

6

That makes sense. Thanks again!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.