Jsonparsefailures parsing apache access logs

Hey guys,

I tried setting up json logging in apache, as demonstrated in the logstash book.

Here's the custom access log definition I'm using:

LogFormat "{ \
  \"host\":\"beta.jokefire.com.jokefire.com\", \
  \"path\":\"/var/log/httpd/jf_beta.jokefire.com_access_log\", \
  \"tags\":[\"Jokefire beta.jokefire.com\"], \
  \"message\": \"%h %l %u %t \\\"%r\\\" %>s %b\", \
  \"timestamp\": \"%{%Y-%m-%dT%H:%M:%S%z}t\", \
  \"clientip\": \"%a\", \
  \"duration\": %D, \
  \"status\": %>s, \
  \"request\": \"%U%q\", \
  \"urlpath\": \"%U\", \
  \"urlquery\": \"%q\", \
  \"method\": \"%m\", \
  \"bytes\": %B, \
  \"vhost\": \"%v\" \
}" beta.jokefire.com_access_json

    CustomLog /var/log/httpd/jf_beta.jokefire.com_access_log beta.jokefire.com_access_json

So my access logs are turning up like this in the actual apache logs on the system:

{       "host":"beta.jokefire.com.jokefire.com",       "path":"/var/log/httpd/jf_beta.jokefire.com_access_log",       "tags":["Jokefire beta.jokefire.com"],       "message": "107.170.139.115 - - [12/Oct/2015:16:35:28 -0400] \"GET /healthcheck.php HTTP/1.1\" 200 5",       "timestamp": "2015-10-12T16:35:28-0400",       "clientip": "107.170.139.115",       "duration": 7995,       "status": 200,       "request": "/healthcheck.php",       "urlpath": "/healthcheck.php",       "urlquery": "",       "method": "GET",       "bytes": 5,       "vhost": "beta.jokefire.com"     }

On the logstash side I'm attempting to use a simple json filter to capture the results:

filter {

  json {
        source => jsonmessage
    }


}

But the results that are turning up in LS are jsonparsefailures:

{
  "_index": "logstash-2015.10.12",
  "_type": "apache_beta_error",
  "_id": "AVBdvbQMy8hMQNvzYdB5",
  "_score": null,
  "_source": {
    "message": "[Thu Oct 08 18:10:21.537414 2015] [authz_core:debug] [pid 980] mod_authz_core.c(809): [client 107.170.139.115:45943] AH01626: authorization result of Require all granted: granted",
    "tags": [
      "_jsonparsefailure"
    ],
    "@version": "1",
    "@timestamp": "2015-10-12T20:28:43.200Z",
    "type": "apache_beta_error",
    "file": "/var/log/httpd/jf_beta.jokefire.com_error_log",
    "host": "ops1",
    "offset": "718959602"
  },
  "fields": {
    "@timestamp": [
      1444681723200
    ]
  },
  "highlight": {
    "type.raw": [
      "@kibana-highlighted-field@apache_beta_error@/kibana-highlighted-field@"
    ],
    "type": [
      "@kibana-highlighted-field@apache_beta_error@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1444681723200
  ]
}

How can I get rid of the jsonparsefailure errors and get this to parse correctly?

Thanks

The message above,

[Thu Oct 08 18:10:21.537414 2015] [authz_core:debug] [pid 980] mod_authz_core.c(809): [client 107.170.139.115:45943] AH01626: authorization result of Require all granted: granted

isn't JSON. That message is from the error log and not the access log. You need to apply the json filter selectively by wrapping it in a conditional. There are examples of that in the documentation.

No. That was from my access log.

 CustomLog /var/log/httpd/jf_beta.jokefire.com_access_log beta.jokefire.com_access_json
    ErrorLog  /var/log/httpd/jf_beta.jokefire.com_error_log

Tailed one line from that log to demonstrate:

[root@ops1:/etc/httpd/conf.d] #tail -1 /var/log/httpd/jf_beta.jokefire.com_access_log
{       "host":"beta.jokefire.com.jokefire.com",       "path":"/var/log/httpd/jf_beta.jokefire.com_access_log",       "tags":["Jokefire beta.jokefire.com"],       "message": "107.170.139.115 - - [12/Oct/2015:16:57:47 -0400] \"GET /healthcheck.php HTTP/1.1\" 200 5",       "timestamp": "2015-10-12T16:57:47-0400",       "clientip": "107.170.139.115",       "duration": 13914,       "status": 200,       "request": "/healthcheck.php",       "urlpath": "/healthcheck.php",       "urlquery": "",       "method": "GET",       "bytes": 5,       "vhost": "beta.jokefire.com"     }

But I'm willing to accept that it might not be JSON. If that's the case, why is my apache custom log definition not working?

I will try wrapping it in a conditional.

Thanks