Parsing problems on logstash + filebeat

Elastic Stack Logstash
1

Hi all,

I've problems for parsing multiline logs with logstash and filebeat:

There my error on logstash:

{:timestamp=>"2016-08-13T05:41:22.216000+0000", :message=>"fetched an invalid config", :config=>" filter {\n  if [type] == \"apache_error\" {\n    grok {\n      pattern => \"%{COMBINEDAPACHELOG}\" \n      add_field => [ \"received_from\", \"%{host}\" ]\n    }\n  }\n    date {\n      match => [ \"timestamp\", \"dd/MMM/yyyy:HH:mm:ss Z\" ]\n  }\n}\n\nfilter {\n  if [type] == \"apache_access\" {\n    grok {\n      match => {\"message\" => \"%{COMBINEDAPACHELOG}\" }\n    }\n    date {\n      match => [ \"timestamp\", \"dd/MMM/yyyy:HH:mm:ss Z\" ]\n }\ngeoip {\n    source => \"clientip\" \n    database => \"/usr/share/GeoIP/GeoLiteCity.dat\" \n    }\n }\n}\n\nfilter {\n  if [type] == \"apache_access_ssl\" {\n    grok {\n      match => {\"message\" => \"%{COMBINEDAPACHELOG}\" }\n    }\n    date {\n      match => [ \"timestamp\", \"dd/MMM/yyyy:HH:mm:ss Z\" ]\n }\ngeoip {\n    source => \"clientip\" \n    database => \"/usr/share/GeoIP/GeoLiteCity.dat\" \n   }\n  }\n} \n\ninput {\n  beats {\n    port => \"5044\"\n    type => \"logs\"\n    congestion_threshold => \"40\"\n    ssl  => true\n    ssl_certificate => \"/etc/okn/filebeat/filebeat.crt\"\n    ssl_key => \"/etc/okn/filebeat/filebeat.key\"\n  }\n}\n\noutput {\n  elasticsearch {\n  #elasticsearch_java {\n  hosts => [\"ubucontrol000:9200\"]\n  #network_host => [\"ubucontrol000:9200\"]\n  #sniffing => true\n  #manage_template => false\n  index => \"%{[@metadata][beat]}-%{+YYYY.MM.dd}\"\n  document_type => \"%{[@metadata][type]}\"\n  user     => \"logstash\"\n  password => \"KeepTheL0gs\"\n  }\n}\n\nfilter {  \n  if [type] == \"syslog\" {\n    grok {\n      match => { \"message\" => \"%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\\[%{POSINT:syslog_pid}\\])?: %{GREEDYDATA:syslog_message}\" }\n      add_field => [ \"received_at\", \"%{@timestamp}\" ]\n      add_field => [ \"received_from\", \"%{host}\" ]\n    }\n    syslog_pri { }\n    date {\n      match => [ \"syslog_timestamp\", \"MMM  d HH:mm:ss\", \"MMM dd HH:mm:ss\" ]\n    }\n  }\n}\n\nif [type] == \"tomcat\" {\n    multiline {\n      pattern => \"TOMCATLOG \\[%{TOMCAT_DATESTAMP:timestamp}\\] \\| %{LOGLEVEL:level} \\| %{JAVACLASS:class} -- %{JAVALOGMESSAGE:logmessage}\"\n      negate => true\n      what => \"previous\"\n    }\n  }\nif \"_grokparsefailure\" in [tags] {\n      drop { }\n    }\n    grok {\n      match => [ \"message\", \"%{TOMCATLOG}\", \"message\", \"%{CATALINALOG}\" ]\n    }\n    date {\n      match => [ \"timestamp\", \"yyyy-MM-dd HH:mm:ss,SSS Z\", \"MMM dd, yyyy HH:mm:ss a\" ]\n    }\n\n", :reason=>"Expected one of #, input, filter, output at line 82, column 1 (byte 1801) after ", :level=>:error}

The logstash configuration for tomcat

if [type] == "tomcat" {
multiline {
pattern => "(^%{TOMCAT_DATESTAMP})|(^%{CATALINA_DATESTAMP})"
negate => true (this is my 82 line)
what => "previous"
}
}
if "_grokparsefailure" in [tags] {
drop { }
}
grok {
match => [ "message", "%{TOMCATLOG}", "message", "%{CATALINALOG}" ]
}
date {
match => [ "timestamp", "yyyy-MM-dd HH:mm:ss,SSS Z", "MMM dd, yyyy HH:mm:ss a" ]
}

I don't see anything wrong.

The patters:

# Tomcat Datestamp
CATALINA_DATESTAMP %{MONTH} %{MONTHDAY}, 20%{YEAR} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) (?:AM|PM)
TOMCAT_DATESTAMP 20%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) %{ISO8601_TIMEZONE}
# Tomcat Logs
CATALINALOG %{CATALINA_DATESTAMP:timestamp} %{JAVACLASS:class} %{JAVALOGMESSAGE:logmessage}
TOMCATLOG %{TOMCAT_DATESTAMP:timestamp} \| %{LOGLEVEL:level} \| %{JAVACLASS:class} - %{JAVALOGMESSAGE:logmessage}
# Java Logs
JAVATHREAD (?:[A-Z]{2}-Processor[\d]+)
JAVACLASS (?:[a-zA-Z0-9-]+\.)+[A-Za-z0-9$]+
JAVAFILE (?:[A-Za-z0-9_.-]+)
JAVASTACKTRACEPART at %{JAVACLASS:class}\.%{WORD:method}\(%{JAVAFILE:file}:%{NUMBER:line}\)
JAVALOGMESSAGE (.*)

And filebeat configuration (a part example)

filebeat

-
  paths:
   - /var/log/tomcat6/catalina.out
  document_type: tomcat
2

The configuration looks okay. Be systematic. Comment out parts of the file and run the file through logstash --configtest until you narrow things down more. You may have garbage characters in there, e.g. straight double quotes that have been turned into smart quotes by a Microsoft product.

1 Like
3

I'm idiot, i saw that the error was in the first line, sure!! i'll need the filter entry .

filter {
if [type] == "tomcat" {
multiline {
pattern => "TOMCATLOG [%{TOMCAT_DATESTAMP:timestamp}] | %{LOGLEVEL:level} | %{JAVACLASS:class} -- %{JAVALOGMESSAGE:logmessage}"
negate => true
what => "previous"
}
if "_grokparsefailure" in [tags] {
drop { }
}
grok {
match => [ "message", "%{TOMCATLOG}", "message", "%{CATALINALOG}" ]
}
date {
match => [ "timestamp", "yyyy-MM-dd HH:mm:ss,SSS Z", "MMM dd, yyyy HH:mm:ss a" ]
}
}
}