Additional field with special data

Elastic Stack Logstash
#1

Hello,
I'm new with elk and want set up an elk stack for varnish logs.
My logstash filter setup looks like this:

filter {
if [type] == "varnish" {
grok {
patterns_dir => ["/etc/logstash/patterns"]
match => {
"message" => [ "(?:%{IPV4:client_ip}|%{IPV4:client_ip}, %{IPV4:lb_ip}) %{USER:user} [%{VARNISHDATE}] "%{WORD:method} %{NOTSPACE:request_page} HTTP/%{NUMBER:http_version}" %{NUMBER:server_response} %{NUMBER:bytes} (?:%{QS:referrer}| -) %{QS:useragent}" ]
}
}
} else ...

Several virtual Hosts are configured at the backend of varnish.

The request_page field contains the protocol and the hole URL string. How do I extract the request Domain from the request_page field and put it in an new field?

#2

The request_page field contains the protocol and the hole URL string. How do I extract the request Domain from the request_page field and put it in an new field?

Do you still want to keep the request_page field?

If yes, use a second grok filter to extract the domain. The standard grok patterns contain a bunch of URL-related patterns that you might find useful. (That said, it's probably possible to extract both the whole request and just the domain in a single expression.)

If no, don't use NOTSPACE; use a more exact pattern that only extracts the domain.

#3

I do this:

                grok {
                        patterns_dir => ["/etc/logstash/patterns"]
                        match => { "message" =>  [ "(?:%{IPV4:client_ip}|%{IPV4:client_ip}, %{IPV4:lb_ip}) %{USER:user} \[%{VARNISHDATE}\] \"%{WORD:method} %{NOTSPACE:request_page} HTTP/%{NUMBER:http_version}\" %{NUMBER:server_response} %{NUMBER:bytes} (?:%{QS:referrer}| -) %{QS:useragent}" ] }
                        match => { "request_page" => "%{URIPROTO}://%{HOSTNAME:domain}" }
                        break_on_match => false
                }

and I will think about mor exact pattern.

Thank you very much

#4