Hello All,
I am very new to the ELK stack and I have been tasked with parsing out some custom logs. I feel like im on the right track but it keeps breaking and Im not sure what Im doing wrong. Basically what I am trying to do is, Ingest a log from a location -> tag the document (document_tag in filebeat.yml) -> ship log to logstash -> filter the incoming log (if[type] == in my filter.conf for logstash) -> parse using custom pattern definitions -> profit. In short i want to parse a specific type of log using my own pattern definitions. I am running a Ubuntu Server 16.04 environment.
In my /var/log/logstash/logstash.log
{:timestamp=>"2017-07-13T08:56:21.720000-0400", :message=>"Pipeline aborted due to error", :exception=>#<Grok::PatternError: pattern %{SHIB_TIMESTAMP} not defined>, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb:123:in `compile'", "org/jruby/RubyKernel.java:1479:in `loop'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb:93:in `compile'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:264:in `register'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:259:in `register'", "org/jruby/RubyHash.java:1342:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:255:in `register'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:182:in `start_workers'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:182:in `start_workers'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:136:in `run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/agent.rb:473:in `start_pipeline'"], :level=>:error}
/etc/logstash/conf.d/42-shib-filter.conf
filter {http://shibboleth.net/ns/profiles/saml2/sso/browser%{SEP}https://shibidp.some.url/idp/shibboleth%{SEP}urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST%{SEP}%{SHIB_TOKEN}%{SEP}%{SHIB_USER_EMAIL} "}
/etc/logstash/conf.d/patterns/extra.txt
SHIB_TOKEN [a-z0-9.-]+ -]+
/etc/filebeat/filebeat.yml
filebeat:
input_type: log
paths:
- /var/log/*
input_type: log
registry_file: /var/lib/filebeat/registry
tatdat
July 13, 2017, 4:54pm
2
Make sure logstash have privileged on patterns folder (use chown command)type in condition in filter logstash, so in filebeat, you should be to use document_type
I did a chmod 0777 on the patterns folder just to see if it would work
Updated filebeat Config:
filebeat:
input_type: log
/var/log/*
input_type: log
/opt/shibboleth-idp/logs/*
How do you test grok success? I used an online debugging tool to create my filter
tatdat
July 13, 2017, 5:02pm
4
Use chown command like this
chown -R logstash. /etc/logstash/conf.d/patterns
You can try to debug grok in here https://grokdebug.herokuapp.com/ Add custom patterns , put your patterns into it.
{:timestamp=>"2017-07-13T08:56:21.720000-0400", :message=>"Pipeline aborted due to error", :exception=>#
This message is damaged so we can't see the error message that normally would follow. Please format the log as preformatted text.
{:timestamp=>"2017-07-13T08:56:21.720000-0400", :message=>"Pipeline aborted due to error", :exception=>#<Grok::PatternError: pattern %{SHIB_TIMESTAMP} not defined>, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb:123:incompile'", "org/jruby/RubyKernel.java:1479:in loop'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb:93:incompile'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:264:in register'", "org/jruby/RubyArray.java:1613:ineach'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:259:in register'", "org/jruby/RubyHash.java:1342:ineach'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:255:in register'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:182:instart_workers'", "org/jruby/RubyArray.java:1613:in each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:182:instart_workers'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:136:in run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/agent.rb:473:instart_pipeline'"], :level=>:error}`
this is how it turns out with the "preformatted" text
Good, so now there's a readable error message. It seems your pattern file isn't loaded. Starting Logstash with --debug can give more clues. In your pattern file, do you have a space after "SHIB_TIMESTAMP" (rather than a tab)?
I decided to forgo that filter and replaced it with %{NOTSPACE:SHIB_DS}
this is the more pressing issue
{:timestamp=>"2017-07-13T15:38:22.030000-0400", :message=>"fetched an invalid config", :config=>"input {\n beats {\n port => 5044\n }\n}\n\nfilter {\n if [type] == \"syslog\" {\n grok {\n match => { \"message\" => \"%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\\[%{POSINT:syslog_pid}\\])?: %{GREEDYDATA:syslog_message}\" }\n add_field => [ \"received_at\", \"%{@timestamp}\" ]\n add_field => [ \"received_from\", \"%{host}\" ]\n }\n syslog_pri { }\n date {\n match => [ \"syslog_timestamp\", \"MMM d HH:mm:ss\", \"MMM dd HH:mm:ss\" ]\n }\n }\n}\n\noutput {\n elasticsearch {\n hosts => [\"localhost:9200\"]\n sniffing => true\n manage_template => false\n index => \"%{[@metadata][beat]}-%{+YYYY.MM.dd}\"\n document_type => \"%{[@metadata][type]}\"\n }\n}\n\nfilter {\n\tif [type] == \"shibboleth-log\" {\n\t\tgrok \n\t\t{\n\t\t\tmatch => { \"message\" => \"%{URIHOST} %{TIME} - %{LOGLEVEL} %{SYSLOG5424SD} - Profile Action %{WORD:LDAP_PROCEDURE}: Login by %{QS:USER_ACCOUNT} %{WORD:STATUS}\"}\n\t\t\tadd_field => {\n\t\t\t\t\"type\" => \"shib-auth-request\"\n\t\t\t}\n\t\t}\n\t\tgrok \n\t\t{\n\t\t\tpatterns_dir => [\"/etc/logstash/conf.d/patterns/]\n\t\t\tmatch => { \"message\" => \"%{URIHOST} %{TIME} - %{LOGLEVEL} %{SYSLOG5424SD} - %{NOTSPACE:SHIB_DS}%{SEP}urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect%{SEP}%{SHIB_TOKEN}%{SEP}%{URI:SHIB_SP}%{SEP}http://shibboleth.net/ns/profiles/saml2/sso/browser%{SEP}https://shibidp.random.url/idp/shibboleth%{SEP}urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST%{SEP}%{SHIB_TOKEN}%{SEP}%{SHIB_USER_EMAIL}\"}\n\t\t\tadd_field => {\n\t\t\t\t\"type\" => \"shib-auth-sp-url\"\n\t\t\t}\n\t\t}\n\t}\n}\n", :reason=>"Expected one of #, {, ,, ] at line 43, column 16 (byte 1069) after filter {\n\tif [type] == \"shibboleth-log\" {\n\t\tgrok \n\t\t{\n\t\t\tmatch => { \"message\" => \"%{URIHOST} %{TIME} - %{LOGLEVEL} %{SYSLOG5424SD} - Profile Action %{WORD:LDAP_PROCEDURE}: Login by %{QS:USER_ACCOUNT} %{WORD:STATUS}\"}\n\t\t\tadd_field => {\n\t\t\t\t\"type\" => \"shib-auth-request\"\n\t\t\t}\n\t\t}\n\t\tgrok \n\t\t{\n\t\t\tpatterns_dir => [\"/etc/logstash/conf.d/patterns/]\n\t\t\tmatch => { \"", :level=>:error}
There's a double quote missing from this line:
patterns_dir => ["/etc/logstash/conf.d/patterns/]
Solving this kind of problem is often done by commenting out parts of the configuration to narrow down the search area.
I feel dumb now lol sometimes it takes a second pair of eyes. Thanks Mr.Back!
system
August 11, 2017, 12:38pm
12
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.