# Grok pattern failing domain.com vs http://domain.com/ **URL:** \ **Category:** Logstash\ **Created:** [October 9, 2017, 10:35pm UTC](https://discuss.elastic.co/t/grok-pattern-failing-domain-com-vs-http-domain-com/103268 "2017-10-09T22:35:54Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![prophoto](https://avatars.discourse-cdn.com/v4/letter/p/f6c823/32.png) [@prophoto](https://discuss.elastic.co/u/prophoto)\ **Post date:** [October 9, 2017, 10:35pm UTC](https://discuss.elastic.co/t/grok-pattern-failing-domain-com-vs-http-domain-com/103268/1 "2017-10-09T22:35:54Z")
I am relatively new to ELK and have gotten pretty far setting up my custom grok pattern for apache logs. I have them slightly customized. I have it 90% of the way there but I'm seeing a failure when the log line has "[domain.com](http://domain.com)" vs "[https://domain.com](https://domain.com)". When the log has the former it fails, but it works fine for the latter. I've had issues in the past where [www.domain.com](http://www.domain.com) and [domain.com](http://domain.com) were being treated as unequal and I am trying to avoid that in my latest build. Log lines. First one fails, second one matches. 12.34.56.78 - - [09/Oct/2017:22:23:00 +0000] [domain.com](http://domain.com) "POST /yt.php HTTP/1.1" 301 240 "[domain.com](http://domain.com)" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36" Server=aws8 "-" 103224 0 12.34.56.78 - - [09/Oct/2017:22:24:45 +0000] [domain2.com](http://domain2.com) "GET /images/icons/32-twitter.png HTTP/1.1" 200 462 "[http://domain2.com/programs/coaching.html](http://domain2.com/programs/coaching.html)" "Mozilla/5.0 (iPhone; CPU iPhone OS 11\_0\_2 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A421 Safari/604.1" Server=aws8 "-" 744 0 Current grok. %{IPORHOST:clientip} - - [%{HTTPDATE:timestamp}] (?:%{IPORHOST:virtualhost}|-) "(?:%{WORD:request\_type} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}|-)" %{NUMBER:response} (?:%{NUMBER:bytes}|-) "(?:%{URI:referrer}|-)" %{QS:agent} Server=(?:%{WORD:server}|-) "(?:-|%{NOTSPACE:ssl\_protocol}|-)" %{BASE10NUM:request\_duration\_ms} %{BASE10NUM:request\_duration\_s} ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 10, 2017, 5:26am UTC](https://discuss.elastic.co/t/grok-pattern-failing-domain-com-vs-http-domain-com/103268/2 "2017-10-10T05:26:40Z")
It appears to be a double-quoted string just like the user agent so why not use the QS pattern? ---
**Author:** ![prophoto](https://avatars.discourse-cdn.com/v4/letter/p/f6c823/32.png) [@prophoto](https://discuss.elastic.co/u/prophoto)\ **Post date:** [October 10, 2017, 1:09pm UTC](https://discuss.elastic.co/t/grok-pattern-failing-domain-com-vs-http-domain-com/103268/3 "2017-10-10T13:09:03Z")
I'm not following? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 10, 2017, 1:47pm UTC](https://discuss.elastic.co/t/grok-pattern-failing-domain-com-vs-http-domain-com/103268/4 "2017-10-10T13:47:53Z")
Use the QS pattern to match the domain name, i.e. replace `"(?:%{URI:referrer}|-)"` with `%{QS:referrer}`. ---
**Author:** ![prophoto](https://avatars.discourse-cdn.com/v4/letter/p/f6c823/32.png) [@prophoto](https://discuss.elastic.co/u/prophoto)\ **Post date:** [October 10, 2017, 4:01pm UTC](https://discuss.elastic.co/t/grok-pattern-failing-domain-com-vs-http-domain-com/103268/5 "2017-10-10T16:01:52Z")
So far so good. Now I'm seeing some log lines with the port included after the virtualhost "[domain.com:80](http://domain.com:80)". Thoughts? 163.172.4.153 - - [10/Oct/2017:15:29:39 +0000] [domain.com:80](http://domain.com:80) "GET /o-neal-drywall-construction-coming-soon.html HTTP/1.1" 200 3369 "[http://domain.com:80/coming-soon.html](http://domain.com:80/coming-soon.html)" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10\_12\_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.81 Safari/537.36" Server=aws8 "-" 269118 0 ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 10, 2017, 4:48pm UTC](https://discuss.elastic.co/t/grok-pattern-failing-domain-com-vs-http-domain-com/103268/6 "2017-10-10T16:48:06Z")
(?:%{IPORHOST:virtualhost}|%{HOSTPORT:virtualhost}|-) perhaps ---
**Author:** ![prophoto](https://avatars.discourse-cdn.com/v4/letter/p/f6c823/32.png) [@prophoto](https://discuss.elastic.co/u/prophoto)\ **Post date:** [October 10, 2017, 6:01pm UTC](https://discuss.elastic.co/t/grok-pattern-failing-domain-com-vs-http-domain-com/103268/7 "2017-10-10T18:01:50Z")
> [@Badger](#): > > (?:%{IPORHOST:virtualhost}|%{HOSTPORT:virtualhost}|-) Works, thanks! ---
**Author:** ![prophoto](https://avatars.discourse-cdn.com/v4/letter/p/f6c823/32.png) [@prophoto](https://discuss.elastic.co/u/prophoto)\ **Post date:** [October 10, 2017, 6:10pm UTC](https://discuss.elastic.co/t/grok-pattern-failing-domain-com-vs-http-domain-com/103268/8 "2017-10-10T18:10:50Z")
Latest: %{IPORHOST:clientip} - - [%{HTTPDATE:timestamp}] (?:%{IPORHOST:virtualhost}|%{HOSTPORT:virtualhost}|-) "(?:%{WORD:request\_type} (?:%{URIPATHPARAM:request}|\*) HTTP/%{NUMBER:httpversion}|-)" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:%{QS:referrer}|-)\ %{QS:agent} Server=(?:%{WORD:server}|-) "(?:-|%{NOTSPACE:ssl\_protocol}|-)" %{BASE10NUM:request\_duration\_ms} %{BASE10NUM:request\_duration\_s} This is failing at HEAD (I think!). 92.239.124.14 - - [10/Oct/2017:18:02:24 +0000] 98.76.54.31:80 "HEAD [http://98.76.54.3:80/mysql/mysqlmanager/](http://98.76.54.3:80/mysql/mysqlmanager/) HTTP/1.1" 404 - "-" "Mozilla/5.0 Jorgee" Server=aws7 "-" 288 0 ![Capture5](https://us1.discourse-cdn.com/elastic/original/3X/8/3/83c591f5f5684dca83bef565881b5189f2f7ff44.JPG) ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 10, 2017, 6:38pm UTC](https://discuss.elastic.co/t/grok-pattern-failing-domain-com-vs-http-domain-com/103268/9 "2017-10-10T18:38:45Z")
It's failing just after HEAD. '[http://98.76.54.3:80/mysql/mysqlmanager/](http://98.76.54.3:80/mysql/mysqlmanager/)' does not match (?:%{URIPATHPARAM:request}|\*), which I do not even think is a valid grok pattern. You really need to make sure what is displayed in the post matches the actual pattern (note the backslash on the square brackets in my version). Use the \ (preformatted text) button. `%{IPORHOST:clientip} - - \[%{HTTPDATE:timestamp}\] (?:%{IPORHOST:virtualhost}|%{HOSTPORT:virtualhost}|-) "(?:%{WORD:request_type} (?:%{URIPATHPARAM:request}|.*) HTTP/%{NUMBER:httpversion}|-)" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:%{QS:referrer}|-)\ %{QS:agent} Server=(?:%{WORD:server}|-) "(?:-|%{NOTSPACE:ssl_protocol}|-)" %{BASE10NUM:request_duration_ms} %{BASE10NUM:request_duration_s}` This matches, but does not capture a named request. You could give that .\* a name using `%{IPORHOST:clientip} - - \[%{HTTPDATE:timestamp}\] (?:%{IPORHOST:virtualhost}|%{HOSTPORT:virtualhost}|-) "(?:%{WORD:request_type} (?:%{URIPATHPARAM:request}|(?.*)) HTTP/%{NUMBER:httpversion}|-)" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:%{QS:referrer}|-)\ %{QS:agent} Server=(?:%{WORD:server}|-) "(?:-|%{NOTSPACE:ssl_protocol}|-)" %{BASE10NUM:request_duration_ms} %{BASE10NUM:request_duration_s}` It is not very pretty, but it does match. ---
**Author:** ![prophoto](https://avatars.discourse-cdn.com/v4/letter/p/f6c823/32.png) [@prophoto](https://discuss.elastic.co/u/prophoto)\ **Post date:** [October 11, 2017, 12:49pm UTC](https://discuss.elastic.co/t/grok-pattern-failing-domain-com-vs-http-domain-com/103268/10 "2017-10-11T12:49:35Z")
You have been very helpful, thank you. I'm down to less than 2% of my log lines in the last 24 hours having a parse failure, so its getting better! The next one I need to tackle is below. It always shows with a 408 request timeout which does not complete the log line and throws an error. I'd like to keep the data so that I can have reports on all error types. How can I get my grok to fill in as much data that is included even if it leaves off the trailing options? xx.xx.xx.xx - - [11/Oct/2017:11:50:00 +0000] "-" 408 - ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 11, 2017, 1:13pm UTC](https://discuss.elastic.co/t/grok-pattern-failing-domain-com-vs-http-domain-com/103268/11 "2017-10-11T13:13:29Z")
Either use a completely different grok expression (a grok filter can list multiple expressions that will be tried in order) or you could make everything after "408 -" optional with `(...)?`. ---
**Author:** ![prophoto](https://avatars.discourse-cdn.com/v4/letter/p/f6c823/32.png) [@prophoto](https://discuss.elastic.co/u/prophoto)\ **Post date:** [October 11, 2017, 1:41pm UTC](https://discuss.elastic.co/t/grok-pattern-failing-domain-com-vs-http-domain-com/103268/12 "2017-10-11T13:41:42Z")
Is that done like below with multiple grok match? grok { match =\> { 'message' =\> ''} match =\> { 'message' =\> ''} } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 11, 2017, 2:21pm UTC](https://discuss.elastic.co/t/grok-pattern-failing-domain-com-vs-http-domain-com/103268/13 "2017-10-11T14:21:04Z")
I'm not sure if that syntax works. There's an example of the supported syntax in the grok filter documentation. ---
**Author:** ![prophoto](https://avatars.discourse-cdn.com/v4/letter/p/f6c823/32.png) [@prophoto](https://discuss.elastic.co/u/prophoto)\ **Post date:** [October 11, 2017, 4:53pm UTC](https://discuss.elastic.co/t/grok-pattern-failing-domain-com-vs-http-domain-com/103268/14 "2017-10-11T16:53:01Z")
Can you point me to the right place? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 12, 2017, 5:20am UTC](https://discuss.elastic.co/t/grok-pattern-failing-domain-com-vs-http-domain-com/103268/15 "2017-10-12T05:20:49Z")
Search for "If you need to match multiple patterns". [https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 9, 2017, 5:21am UTC](https://discuss.elastic.co/t/grok-pattern-failing-domain-com-vs-http-domain-com/103268/16 "2017-11-09T05:21:05Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.