# Custom patterns using patterns\_dir is not working

**URL:** <https://discuss.elastic.co/t/custom-patterns-using-patterns-dir-is-not-working/118572>\
**Category:** Logstash\
**Created:** [February 6, 2018, 7:36am UTC](https://discuss.elastic.co/t/custom-patterns-using-patterns-dir-is-not-working/118572 "2018-02-06T07:36:01Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ranganath\_nangineni](https://avatars.discourse-cdn.com/v4/letter/r/3ec8ea/32.png) [@ranganath\_nangineni](https://discuss.elastic.co/u/ranganath_nangineni)\
**Post date:** [February 6, 2018, 7:36am UTC](https://discuss.elastic.co/t/custom-patterns-using-patterns-dir-is-not-working/118572/1 "2018-02-06T07:36:01Z")

</div>

I have the below custom patterns under /usr/share/logstash/patterns and /etc/logstash/patterns directories. These were created to accept HTTPDUSER in the COMBINEDPATCHELOG pattern.

The pattern is as below

USERNAME [a-zA-Z0-9._-]+  
USER %{USERNAME}  
EMAILLOCALPART [a-zA-Z][a-zA-Z0-9_.+-=:]+  
EMAILADDRESS %{EMAILLOCALPART}@%{HOSTNAME}  
HTTPDUSER %{EMAILADDRESS}|%{USER}  
COMMONAPACHELOG %{IPORHOST:clientip} %{HTTPDUSER:ident} %{HTTPDUSER:auth} [%{HTTPDATE:timestamp}] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)  
COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}  
SIMPLEAPACHELOG %{COMBINEDAPACHELOG} "%{IP:trueclientip}" "%{GREEDYDATA:filetype}"

# Grok filter is as below:

if [fields][app] == "simple" {

```
    grok {
       patterns_dir => ["/usr/share/logstash/patterns" , "/etc/logstash/patterns"]
       match => { "message" => "%{COMBINEDAPACHELOG} "%{IP:trueclientip}" "%{QS:filetype}" "}
       #match => { "message" => "%{SIMPLEAPACHELOG}" }
       #match => { "message" => "%{COMBINEDAPACHELOG}" }
    }

```

But it is not accepting the format. I doubts it is not even looking into the patterns directory / overlooked by the existing default patterns.

# Patterns core plugin:

logstash-patterns-core (4.1.2)  
logstash-filter-grok (3.4.3)

The same pattern worked fine with the [grokconstructor.appspot.com/](http://grokconstructor.appspot.com/) [https://grokdebug.herokuapp.com/](https://grokdebug.herokuapp.com/)

# Log Pattern:

151.71.120.51 - [Jacob@gmail.com](mailto:Jacob@gmail.com) [23/Jan/2018:22:35:51 -0800] "GET /content/resources/layouts/Trays/Sample/menu\_bar\_divider.gif HTTP/1.1" 200 1234 "[https://Sample.xyz.com/site/mktg/gic/ITK/index.html](https://Sample.xyz.com/site/mktg/gic/ITK/index.html)" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" "10.17.158.213" "image/gif"

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 6, 2018, 8:32am UTC](https://discuss.elastic.co/t/custom-patterns-using-patterns-dir-is-not-working/118572/2 "2018-02-06T08:32:44Z")

</div>

> ```
> match => { "message" => "%{COMBINEDAPACHELOG} "%{IP:trueclientip}" "%{QS:filetype}" "}
> 
> ```

Multiple problems:

- You can't have double quotes inside a double quoted string. Make the string single-quoted.

- Remove the trailing space.

- The QS pattern includes the surrounding double quotes so unless you want to match `""foo""` you should say `%{QS}` instead of `"%{QS}"`.

---

<div class="post-metadata">

**Author:** ![ranganath\_nangineni](https://avatars.discourse-cdn.com/v4/letter/r/3ec8ea/32.png) [@ranganath\_nangineni](https://discuss.elastic.co/u/ranganath_nangineni)\
**Post date:** [February 6, 2018, 8:52am UTC](https://discuss.elastic.co/t/custom-patterns-using-patterns-dir-is-not-working/118572/3 "2018-02-06T08:52:42Z")

</div>

I have placed the below pattern in the pattern file and then tried this pattern in Grok,

SIMPLEAPACHELOG %{IPORHOST:clientip} %{HTTPDUSER:ident} %{HTTPDUSER:auth} [%{HTTPDATE:timestamp}] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} "%{IP:trueclientip}" "%{GREEDYDATA:filetype}"

But it is not parsing the trueclientip and the filetype.

grok {  
patterns\_dir =\> ["/usr/share/logstash/patterns" , "/etc/logstash/patterns"]  
#match =\> { "message" =\> "%{COMBINEDAPACHELOG} "%{IP:trueclientip}" "%{QS:filetype}" "}  
match =\> { "message" =\> "%{SIMPLEAPACHELOG}" }  
#match =\> { "message" =\> "%{COMBINEDAPACHELOG}" }  
}

so the ' (single quote ) will work as per your input ?

match =\> { "message" =\> '%{COMBINEDAPACHELOG} "%{IP:trueclientip}" %{QS:filetype}'}

> [@magnusbaeck](#):
>
> '%{COMBINEDAPACHELOG} "%{IP:trueclientip}" %{QS:filetype}'

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 20, 2018, 8:53am UTC](https://discuss.elastic.co/t/custom-patterns-using-patterns-dir-is-not-working/118572/4 "2018-02-20T08:53:07Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.