### Author:  [@thecrimsoncoder](https://discuss.elastic.co/u/thecrimsoncoder)
#### Post date: [July 13, 2017, 2:28pm UTC](https://discuss.elastic.co/t/logstash-custom-filters-and-patterns/93031/1 "2017-07-13T14:28:14Z")
Hello All,
I am very new to the ELK stack and I have been tasked with parsing out some custom logs. I feel like im on the right track but it keeps breaking and Im not sure what Im doing wrong. Basically what I am trying to do is, Ingest a log from a location -\> tag the document (document\_tag in filebeat.yml) -\> ship log to logstash -\> filter the incoming log (if[type] == in my filter.conf for logstash) -\> parse using custom pattern definitions -\> profit. In short i want to parse a specific type of log using my own pattern definitions. I am running a Ubuntu Server 16.04 environment.
In my /var/log/logstash/logstash.log
> {:timestamp=\>"2017-07-13T08:56:21.720000-0400", :message=\>"Pipeline aborted due to error", :exception=\>#\, :backtrace=\>["/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb:123:in `compile'", "org/jruby/RubyKernel.java:1479:in `loop'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb:93:in `compile'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:264:in `register'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:259:in `register'", "org/jruby/RubyHash.java:1342:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:255:in `register'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:182:in `start_workers'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:182:in `start_workers'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:136:in `run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/agent.rb:473:in `start\_pipeline'"], :level=\>:error}
> {:timestamp=\>"2017-07-13T08:56:24.732000-0400", :message=\>"stopping pipeline", :id=\>"main"}
/etc/logstash/conf.d/42-shib-filter.conf
> filter {
> if [type] == "shibboleth-log" {
> grok
> {
> match =\> { "message" =\> "%{URIHOST} %{TIME} - %{LOGLEVEL} %{SYSLOG5424SD} - Profile Action %{WORD:LDAP\_PROCEDURE}: Login by %{QS:USER\_ACCOUNT} %{WORD:STATUS}"}
> add\_field =\> {
> "type" =\> "shib-auth-request"
> }
> }
> grok
> {
> patterns\_dir =\> ["/etc/logstash/conf.d/patterns/]
> match =\> { "message" =\> "%{URIHOST} %{TIME} - %{LOGLEVEL} %{SYSLOG5424SD} - %{SHIB\_TIMESTAMP}%{SEP}urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect%{SEP}%{SHIB\_TOKEN}%{SEP}%{URI:SHIB\_SP}%{SEP}[http://shibboleth.net/ns/profiles/saml2/sso/browser%{SEP}https://shibidp.some.url/idp/shibboleth%{SEP}urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST%{SEP}%{SHIB\_TOKEN}%{SEP}%{SHIB\_USER\_EMAIL}](http://shibboleth.net/ns/profiles/saml2/sso/browser%25%7BSEP%7Dhttps://shibidp.some.url/idp/shibboleth%25%7BSEP%7Durn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST%25%7BSEP%7D%25%7BSHIB_TOKEN%7D%25%7BSEP%7D%25%7BSHIB_USER_EMAIL%7D)"}
> add\_field =\> {
> "type" =\> "shib-auth-sp-url"
> }
> }
> }
> }
/etc/logstash/conf.d/patterns/extra.txt
> SHIB\_TOKEN [a-z0-9._-]+
> SHIB\_TIMESTAMP [a-zA-Z0-9._-]+
> SEP [|.]
> SHIB\_USER\_EMAIL [a-zA-Z0-9.@]+
/etc/filebeat/filebeat.yml
> ## filebeat: prospectors:
>
> ```
> input_type: log
> paths:
> - /var/log/*
>
> ```
>
> - input\_type: log
> paths:
> - /opt/shibboleth-idp/logs/\*
> registry\_file: /var/lib/filebeat/registry
> output:
> logstash:
> # The Logstash hosts
> hosts: ["localhost:5044"]
---
### Author:  [@tatdat](https://discuss.elastic.co/u/tatdat)
#### Post date: [July 13, 2017, 4:54pm UTC](https://discuss.elastic.co/t/logstash-custom-filters-and-patterns/93031/2 "2017-07-13T16:54:31Z")
Make sure logstash have privileged on `patterns` folder (use chown command)
I saw you used `type` in condition in filter logstash, so in filebeat, you should be to use `document_type`
And you should be test grok success before use pattern.
---
### Author:  [@thecrimsoncoder](https://discuss.elastic.co/u/thecrimsoncoder)
#### Post date: [July 13, 2017, 4:57pm UTC](https://discuss.elastic.co/t/logstash-custom-filters-and-patterns/93031/3 "2017-07-13T16:57:48Z")
I did a chmod 0777 on the patterns folder just to see if it would work
Updated filebeat Config:
> filebeat:
> prospectors:
>
> - input\_type: log
> paths:
> - /var/log/\*
> document\_type: syslog
>
> - input\_type: log
> paths:
> - /opt/shibboleth-idp/logs/\*
> document\_type: shibboleth-log
How do you test grok success? I used an online debugging tool to create my filter
---
### Author:  [@tatdat](https://discuss.elastic.co/u/tatdat)
#### Post date: [July 13, 2017, 5:02pm UTC](https://discuss.elastic.co/t/logstash-custom-filters-and-patterns/93031/4 "2017-07-13T17:02:49Z")
Use chown command like this
> chown -R logstash. /etc/logstash/conf.d/patterns
> [@thecrimsoncoder](#):
>
> How do you test grok success? I used an online debugging tool to create my filter
You can try to debug grok in here [https://grokdebug.herokuapp.com/](https://grokdebug.herokuapp.com/)
Use `Add custom patterns `, put your patterns into it.
---
### Author:  [@thecrimsoncoder](https://discuss.elastic.co/u/thecrimsoncoder)
#### Post date: [July 13, 2017, 5:10pm UTC](https://discuss.elastic.co/t/logstash-custom-filters-and-patterns/93031/5 "2017-07-13T17:10:33Z")
thats the tool i used, [it compiles without any errors](http://imgur.com/a/PxIgR)
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [July 13, 2017, 7:20pm UTC](https://discuss.elastic.co/t/logstash-custom-filters-and-patterns/93031/6 "2017-07-13T19:20:46Z")
> {:timestamp=\>"2017-07-13T08:56:21.720000-0400", :message=\>"Pipeline aborted due to error", :exception=\>#
This message is damaged so we can't see the error message that normally would follow. Please format the log as preformatted text.
---
### Author:  [@thecrimsoncoder](https://discuss.elastic.co/u/thecrimsoncoder)
#### Post date: [July 13, 2017, 7:27pm UTC](https://discuss.elastic.co/t/logstash-custom-filters-and-patterns/93031/7 "2017-07-13T19:27:22Z")
`{:timestamp=>"2017-07-13T08:56:21.720000-0400", :message=>"Pipeline aborted due to error", :exception=>#, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb:123:in`compile'", "org/jruby/RubyKernel.java:1479:in `loop'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb:93:in`compile'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:264:in `register'", "org/jruby/RubyArray.java:1613:in`each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:259:in `register'", "org/jruby/RubyHash.java:1342:in`each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:255:in `register'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:182:in`start\_workers'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:182:in`start\_workers'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:136:in `run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/agent.rb:473:in`start\_pipeline'"], :level=\>:error}`
this is how it turns out with the "preformatted" text
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [July 13, 2017, 7:35pm UTC](https://discuss.elastic.co/t/logstash-custom-filters-and-patterns/93031/8 "2017-07-13T19:35:28Z")
Good, so now there's a readable error message. It seems your pattern file isn't loaded. Starting Logstash with `--debug` can give more clues. In your pattern file, do you have a space after "SHIB\_TIMESTAMP" (rather than a tab)?
---
### Author:  [@thecrimsoncoder](https://discuss.elastic.co/u/thecrimsoncoder)
#### Post date: [July 13, 2017, 7:40pm UTC](https://discuss.elastic.co/t/logstash-custom-filters-and-patterns/93031/9 "2017-07-13T19:40:52Z")
I decided to forgo that filter and replaced it with %{NOTSPACE:SHIB\_DS}
this is the more pressing issue
`{:timestamp=>"2017-07-13T15:38:22.030000-0400", :message=>"fetched an invalid config", :config=>"input {\n beats {\n port => 5044\n }\n}\n\nfilter {\n if [type] == \"syslog\" {\n grok {\n match => { \"message\" => \"%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\\[%{POSINT:syslog_pid}\\])?: %{GREEDYDATA:syslog_message}\" }\n add_field => [\"received_at\", \"%{@timestamp}\"]\n add_field => [\"received_from\", \"%{host}\"]\n }\n syslog_pri { }\n date {\n match => [\"syslog_timestamp\", \"MMM d HH:mm:ss\", \"MMM dd HH:mm:ss\"]\n }\n }\n}\n\noutput {\n elasticsearch {\n hosts => [\"localhost:9200\"]\n sniffing => true\n manage_template => false\n index => \"%{[@metadata][beat]}-%{+YYYY.MM.dd}\"\n document_type => \"%{[@metadata][type]}\"\n }\n}\n\nfilter {\n\tif [type] == \"shibboleth-log\" {\n\t\tgrok \n\t\t{\n\t\t\tmatch => { \"message\" => \"%{URIHOST} %{TIME} - %{LOGLEVEL} %{SYSLOG5424SD} - Profile Action %{WORD:LDAP_PROCEDURE}: Login by %{QS:USER_ACCOUNT} %{WORD:STATUS}\"}\n\t\t\tadd_field => {\n\t\t\t\t\"type\" => \"shib-auth-request\"\n\t\t\t}\n\t\t}\n\t\tgrok \n\t\t{\n\t\t\tpatterns_dir => [\"/etc/logstash/conf.d/patterns/]\n\t\t\tmatch => { \"message\" => \"%{URIHOST} %{TIME} - %{LOGLEVEL} %{SYSLOG5424SD} - %{NOTSPACE:SHIB_DS}%{SEP}urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect%{SEP}%{SHIB_TOKEN}%{SEP}%{URI:SHIB_SP}%{SEP}http://shibboleth.net/ns/profiles/saml2/sso/browser%{SEP}https://shibidp.random.url/idp/shibboleth%{SEP}urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST%{SEP}%{SHIB_TOKEN}%{SEP}%{SHIB_USER_EMAIL}\"}\n\t\t\tadd_field => {\n\t\t\t\t\"type\" => \"shib-auth-sp-url\"\n\t\t\t}\n\t\t}\n\t}\n}\n", :reason=>"Expected one of #, {, ,, ] at line 43, column 16 (byte 1069) after filter {\n\tif [type] == \"shibboleth-log\" {\n\t\tgrok \n\t\t{\n\t\t\tmatch => { \"message\" => \"%{URIHOST} %{TIME} - %{LOGLEVEL} %{SYSLOG5424SD} - Profile Action %{WORD:LDAP_PROCEDURE}: Login by %{QS:USER_ACCOUNT} %{WORD:STATUS}\"}\n\t\t\tadd_field => {\n\t\t\t\t\"type\" => \"shib-auth-request\"\n\t\t\t}\n\t\t}\n\t\tgrok \n\t\t{\n\t\t\tpatterns_dir => [\"/etc/logstash/conf.d/patterns/]\n\t\t\tmatch => { \"", :level=>:error}`
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [July 13, 2017, 8:11pm UTC](https://discuss.elastic.co/t/logstash-custom-filters-and-patterns/93031/10 "2017-07-13T20:11:03Z")
There's a double quote missing from this line:
```
patterns_dir => ["/etc/logstash/conf.d/patterns/]
```
Solving this kind of problem is often done by commenting out parts of the configuration to narrow down the search area.
---
### Author:  [@thecrimsoncoder](https://discuss.elastic.co/u/thecrimsoncoder)
#### Post date: [July 14, 2017, 12:38pm UTC](https://discuss.elastic.co/t/logstash-custom-filters-and-patterns/93031/11 "2017-07-14T12:38:13Z")
I feel dumb now lol sometimes it takes a second pair of eyes. Thanks Mr.Back!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 11, 2017, 12:38pm UTC](https://discuss.elastic.co/t/logstash-custom-filters-and-patterns/93031/12 "2017-08-11T12:38:55Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.