Logstash apache access referrer value parsing

Elastic Stack Logstash
1

Hello,

I'm trying to create visualization on Kibana based upon referrer value from apache logs. Below is a sample log from apache access logs.

127.0.0.1 - - [20/May/2016:00:00:14 +0000] "GET /secure/queue/handleQueueIndex.do HTTP/1.1" 200 7127 1232 "http://localhost:8080/secure/queue/handleQueueIndex.do" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/7.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)"

Grok pattern used:
%{COMMONAPACHELOG} %{NUMBER:time} %{QS:referrer} %{QS:agent}

Output based upon grok pattern:
"_source": {
"referrer": ""https://localhost:8080/secure/queue/handleQueueIndex.do"",
}

I want to parse referrer value and define mapping from it like below. Then create visualization on kibana based on timestamp, count and mapping values.

E.g (Key value pair)
/secure/queue -> queue manager
/secure/rules -> rule manager

Can someone help me in solving this problem or any example link?

2

Use a grok filter to extract the interesting parts of the URL from the referrer field and use a translate filter to transform those URL fragments to the human-readable strings you want to display in Kibana.

3

I was trying something like this using mutate and gsub but it's not working I guess. Can you please give me an example on how to do using grok filter and translate as I'm new to ELK

filter {
grok {
match => { "message" => "%{COMMONAPACHELOG} %{NUMBER:time} %{QS:referrer} %{QS:agent}"}
add_field => [ "requestURL", "%{referrer}" ]
}
mutate {
gsub => ["requestURL", "^./secure/queue.$", "queue module",
"requestURL", "^./secure/rules.$", "rule module"
]
}

4

The following grok filter should extract the first two path components of the requestURL field into a path field:

grok {
  match => [
    "requestURL",
    '^"%{URIPROTO}://%{URIHOST}(?<path>/[^/]+/[^/]+)/',
  ]
}

Then use the translate filter to map the contents of the path field to whatever you like.