I marked as solved becuase what I changed based on your answer did work, but I think I did something wrong as now I am getting 4000 logs per minute in elasticsearch rather than my expected 200...
I did this:
input {
beats {
port => "5044"
}
}
filter {
grok {
match => { "message" =>"%{IP:client_ip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:time_stamp}\] %{QS:web_site} %{NUMBER:server_port} \"%{WORD:request_method} %{URIPATHPARAM:uri_path} HTTP/%{NUMBER:http_version}\" %{NUMBER:response} (?:%{QS:referer}|-) %{QS:user_agent} %{NUMBER:bytes_received} %{NUMBER:bytes_sent}"}
match => { "message" =>"%{IP:client_ip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:time_stamp}\] %{QS:web_site} %{NUMBER:server_port} %{QS:request_method} %{NUMBER:response} %{QS:referer} %{QS:user_agent} %{NUMBER:bytes_received} %{NUMBER:bytes_sent}" }
}
geoip {
source => "client_ip"
}
}
output {
elasticsearch {
hosts => [ "10.1.0.20:9200" ]
}
}
Which, re-reading your post, does not look like what you actually suggested. Can you give me a better example of what I would do in my case?