Eliminating _grokparsefailure

Elastic Stack Logstash
1

Hello, all.

I've been working on setting up my ELK stack for about the past two weeks. I must admit that I've cobbled together my instance based on various posts, how-to's, etc. Because of that, my configuration is probably a mess. That notwithstanding, my instance does work; I'm getting log data in Kibana from Windows, Linux, and ESXi (more on ESXi in another post here) servers.

However, for every single log event I'm capturing, I see _grokparsefailure in tags. I've read about, and tried, everything I can, but no joy. I can't get rid of _grokparsefailure.

I know this is a very lot to ask of you all, but I've posted all of my config files, including from logstash-forwarder and nxlog, here: http://pastebin.com/4hDae6bT. I would be most grateful if someone would take a look, and see where I've possibly created a condition that creates the _grokparsefailures. Or, to suggest another means by which I can get rid of them.

With thanks,

Diggy

2

At least one cause of the _grokparsefailure tags everywhere is the fact that the grok filter with COMBINEDAPACHELOG is applied to all messages even though it'll obviously only apply to HTTP logs. Secondly you should probably match against the message field instead of line. This should work better:

if [type] == "apache-access" {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
 
  date {
    match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
    locale => "en"
  }
}

You might have other filters that should be wrapped in conditionals. These two were the most obvious ones.

3

Magnus,

Once again, you've helped solve the problem. Thank you so much.

Interestingly, syslog-generated logs from my Linux hosts don't contain the field "message" but, rather, "line". Thus, "match against the message field instead of line" wouldn't work. I don't know why that is. Any idea?

Diggy

4

logstash-forwarder sends the payload from a log file in the line field but Logstash itself has standardized on message.

You should drop logstash-forwarder in favor of Filebeat since the former is deprecated and not maintained.

5

Hi, Magnus.

Just an update that I've installed, and am successfully using, Filebeat in my Linux servers. Of course, I'm now seeing the "message" field, and have made the requisite changes in my logstash conf files.

Diggy