Hello,
I'm new with elk and want set up an elk stack for varnish logs.
My logstash filter setup looks like this:
filter {
if [type] == "varnish" {
grok {
patterns_dir => ["/etc/logstash/patterns"]
match => {
"message" => [ "(?:%{IPV4:client_ip}|%{IPV4:client_ip}, %{IPV4:lb_ip}) %{USER:user} [%{VARNISHDATE}] "%{WORD:method} %{NOTSPACE:request_page} HTTP/%{NUMBER:http_version}" %{NUMBER:server_response} %{NUMBER:bytes} (?:%{QS:referrer}| -) %{QS:useragent}" ]
}
}
} else ...
Several virtual Hosts are configured at the backend of varnish.
The request_page field contains the protocol and the hole URL string. How do I extract the request Domain from the request_page field and put it in an new field?
The request_page field contains the protocol and the hole URL string. How do I extract the request Domain from the request_page field and put it in an new field?
Do you still want to keep the request_page
field?
If yes, use a second grok filter to extract the domain. The standard grok patterns contain a bunch of URL-related patterns that you might find useful. (That said, it's probably possible to extract both the whole request and just the domain in a single expression.)
If no, don't use NOTSPACE; use a more exact pattern that only extracts the domain.
I do this:
grok {
patterns_dir => ["/etc/logstash/patterns"]
match => { "message" => [ "(?:%{IPV4:client_ip}|%{IPV4:client_ip}, %{IPV4:lb_ip}) %{USER:user} \[%{VARNISHDATE}\] \"%{WORD:method} %{NOTSPACE:request_page} HTTP/%{NUMBER:http_version}\" %{NUMBER:server_response} %{NUMBER:bytes} (?:%{QS:referrer}| -) %{QS:useragent}" ] }
match => { "request_page" => "%{URIPROTO}://%{HOSTNAME:domain}" }
break_on_match => false
}
and I will think about mor exact pattern.
Thank you very much