Responsetime doesnt come in JSON object in case of %{COMMONAPACHELOG}"

Elastic Stack Logstash
#1

Hi,
I am using logstash 1.5.0 as indexer to filter apache access logs.
I use match => { "message" => "%{COMMONAPACHELOG}" } in my groke filter.

Sample log is :

xx.xx.xx.xxx - - [24/Feb/2016:07:45:54 +0200] "GET /serviceprovider/id/101 HTTP/1.1" 500 2267 28

JSON is:

{
  "_index": "logstash-2016.02.24",
  "_type": "access-logs",
  "_id": "AVMScwRBC7tfwD5zOg5k",
  "_score": null,
  "_source": {
    "message": "xx.xx.xx.xxx - - [24/Feb/2016:07:45:54 +0200] \"GET /serviceprovider/id/101 HTTP/1.1\" 500 2267 28",
    "@version": "1",
    "@timestamp": "2016-02-24T08:44:00.664Z",
    "type": "access-logs",
    "host": "HostName",
    "path": "localhost-access.log",
    "clientip": "xx.xx.xx.xxx",
    "ident": "-",
    "auth": "-",
    "timestamp": "24/Feb/2016:07:45:54 +0200",
    "verb": "GET",
    "request": "/serviceprovider/id/101",
    "httpversion": "1.1",
    "response": "500",
    "bytes": "2267"
  },
  "fields": {
    "@timestamp": [
      1456303440664
    ]
  },
  "sort": [
    1456303440664
  ]
}

I am expecting response time which is th elast field should be splited and added in json docuemnt, but it doesn't.
Also there is not any _grokeParseFailure error.
Please guide which format should be used instead of COMMONAPACHELOG

br,
Sunil Chaudhari

#2

I am expecting response time which is th elast field should be splited and added in json docuemnt, but it doesn't.

Why would you expect that? The response time isn't included in the Apache common log format.

Please guide which format should be used instead of COMMONAPACHELOG

Try this:

%{COMMONAPACHELOG} %{INT:response_time:int}

You should use the mutate filter's convert option to convert the type of at least the bytes field to an integer. Otherwise you won't be able to perform numerical aggregations in e.g. Kibana.

#3

Hi Magnus,
It doesnt work. It tags _grokeParseFailure .

br,
Sunil

#4

Hi,
It works when I give proper unit in miliseconds:

%{INT:response_ms:int}

Thanks a lot! :slight_smile:

#5

Uh, what? The "_ms" suffix can't possibly have anything to do with this. It must've been something else. Going back to your original sample message,

xx.xx.xx.xxx - - [24/Feb/2016:07:45:54 +0200] \"GET /serviceprovider/id/101 HTTP/1.1\" 500 2267 28

do you really have backslashes before the double quotes in your log? If so that's why COMMONAPACHELOG doesn't work.

#6

Hi,
I have tested this twice and I am sure it is because of time units.
About backslash, its not there, it might be due to manipulation of input log file.

br,
Sunil

#7