# Need help configuring my grok filters **URL:** \ **Category:** Logstash\ **Created:** [March 21, 2017, 8:53pm UTC](https://discuss.elastic.co/t/need-help-configuring-my-grok-filters/79500 "2017-03-21T20:53:28Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Rich\_Johnson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rich_johnson/32/16129_2.png) [@Rich\_Johnson](https://discuss.elastic.co/u/Rich_Johnson)\ **Post date:** [March 21, 2017, 8:53pm UTC](https://discuss.elastic.co/t/need-help-configuring-my-grok-filters/79500/1 "2017-03-21T20:53:28Z")
As I am only starting to test with ElasticStack, I have only configured 3 hosts with filebeat installed to forward apache logs to logstash. My grok filter looks like this: ```auto input { beats { port => "5044" } } filter { grok { match => { "message" =>"%{IP:client_ip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:time_stamp}\] %{QS:web_site} %{NUMBER:server_port} \"%{WORD:request_method} %{URIPATHPARAM:uri_path} HTTP/%{NUMBER:http_version}\" %{NUMBER:response} (?:%{QS:referer}|-) %{QS:user_agent} %{NUMBER:bytes_received} %{NUMBER:bytes_sent}"} } geoip { source => "client_ip" } } output { if "_grokparsefailure" in [tags] { file { path => "var/log/logstash/failed/failed_apache_events-%{+YYYY-MM-dd}" } } elasticsearch { hosts => ["10.1.0.20:9200"] } } ``` As you can see, I've configured it to output the log to a failed log if it contains \_grokparsefailure. There is only one format of log that gets sent to the failed log, but it happens over 20,000 times per day and it looks like this: ```auto 67.171.49.122 - - [09/Mar/2017:07:18:45 -0700] "-" 443 "-" 408 "-" "-" 568 137 ``` I have build a filter that correctly matches up each field like so: ```auto %{IP:client_ip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:time_stamp}\] %{QS:web_site} %{NUMBER:server_port} %{QS:request_method} %{NUMBER:response} %{QS:referer} %{QS:user_agent} %{NUMBER:bytes_received} %{NUMBER:bytes_sent} ``` but I do not know how to say something like, if the log contains \_grokparsefailure, then try this filter". ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 22, 2017, 6:24am UTC](https://discuss.elastic.co/t/need-help-configuring-my-grok-filters/79500/2 "2017-03-22T06:24:22Z")
A grok filter can be given multiple expressions. They will be tried in order, first match wins. ``` match => { "message" => ["expression1", "expression"] } ``` ---
**Author:** ![Rich\_Johnson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rich_johnson/32/16129_2.png) [@Rich\_Johnson](https://discuss.elastic.co/u/Rich_Johnson)\ **Post date:** [March 22, 2017, 11:47pm UTC](https://discuss.elastic.co/t/need-help-configuring-my-grok-filters/79500/3 "2017-03-22T23:47:22Z")
I marked as solved becuase what I changed based on your answer did work, but I think I did something wrong as now I am getting 4000 logs per minute in elasticsearch rather than my expected 200... I did this: ```auto input { beats { port => "5044" } } filter { grok { match => { "message" =>"%{IP:client_ip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:time_stamp}\] %{QS:web_site} %{NUMBER:server_port} \"%{WORD:request_method} %{URIPATHPARAM:uri_path} HTTP/%{NUMBER:http_version}\" %{NUMBER:response} (?:%{QS:referer}|-) %{QS:user_agent} %{NUMBER:bytes_received} %{NUMBER:bytes_sent}"} match => { "message" =>"%{IP:client_ip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:time_stamp}\] %{QS:web_site} %{NUMBER:server_port} %{QS:request_method} %{NUMBER:response} %{QS:referer} %{QS:user_agent} %{NUMBER:bytes_received} %{NUMBER:bytes_sent}" } } geoip { source => "client_ip" } } output { elasticsearch { hosts => ["10.1.0.20:9200"] } } ``` Which, re-reading your post, does not look like what you actually suggested. Can you give me a better example of what I would do in my case? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 23, 2017, 6:00am UTC](https://discuss.elastic.co/t/need-help-configuring-my-grok-filters/79500/4 "2017-03-23T06:00:36Z")
> I marked as solved becuase what I changed based on your answer did work, but I think I did something wrong as now I am getting 4000 logs per minute in elasticsearch rather than my expected 200... Unless you actually have that amount of traffic I can think of two reasons: - You have a clone filter in your configuration (which I don't think you have). - You accidentally have multiple copies of your elasticsearch output. Logstash reads _all_ configuration files in /etc/logstash/conf.d (or whatever directory you tell it to read). Your grok filter setup with two `match` settings probably works but I prefer using the documented syntax in my previous example. ---
**Author:** ![Rich\_Johnson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rich_johnson/32/16129_2.png) [@Rich\_Johnson](https://discuss.elastic.co/u/Rich_Johnson)\ **Post date:** [March 23, 2017, 5:40pm UTC](https://discuss.elastic.co/t/need-help-configuring-my-grok-filters/79500/5 "2017-03-23T17:40:18Z")
I don't have any other filters in my conf.d directory. All I know is, at point #1 I added the second "match" statement that was supposed to catch all the logs that had a \_grokparsefailure. At point #2 is when I raised the issue on this forum. At point #3 I removed the second "match" statement. Maybe Elasticsearch was going back through all those logs tagged as \_grokparsefailure and re-indexing them according to the new filter? ![](https://us1.discourse-cdn.com/elastic/original/3X/8/9/893ce162e97323b2953eb4be527f31c38c0fe039.png) Can you give a more thorough example of your preferred approach using my filter above? I don't quite understand... ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 23, 2017, 6:02pm UTC](https://discuss.elastic.co/t/need-help-configuring-my-grok-filters/79500/6 "2017-03-23T18:02:09Z")
> I don't have any other filters in my conf.d directory. All I know is, ... Hmm. Maybe specifying two `match` options works differently than I thought. > Maybe Elasticsearch was going back through all those logs tagged as \_grokparsefailure and re-indexing them according to the new filter? No, it won't do that by itself. > Can you give a more thorough example of your preferred approach using my filter above? I don't quite understand... ```plaintext grok { match => { "message" => [ "%{IP:client_ip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:time_stamp}\] %{QS:web_site} %{NUMBER:server_port} \"%{WORD:request_method} %{URIPATHPARAM:uri_path} HTTP/%{NUMBER:http_version}\" %{NUMBER:response} (?:%{QS:referer}|-) %{QS:user_agent} %{NUMBER:bytes_received} %{NUMBER:bytes_sent}", "%{IP:client_ip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:time_stamp}\] %{QS:web_site} %{NUMBER:server_port} %{QS:request_method} %{NUMBER:response} %{QS:referer} %{QS:user_agent} %{NUMBER:bytes_received} %{NUMBER:bytes_sent}" ] } } ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 20, 2017, 6:02pm UTC](https://discuss.elastic.co/t/need-help-configuring-my-grok-filters/79500/7 "2017-04-20T18:02:23Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.