AWS ELK : Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError"

Elastic Stack Logstash
1

Hi All,
Getting LogStash::ConfigurationError in AWS ELK environment.
below is my logstash file

input {
      file {
            type => "audit"
            path => ["/var/log/tomcat8/**/*-audit.log"]
            start_position => "beginning"
            sincedb_path => "/dev/null"
            codec => multiline {
            pattern => "^%{TIMESTAMP_ISO8601} "
            negate => true
            what => previous
            }
      }

      file {
            type => "app"
            path => ["/var/log/tomcat8/**/*-info.log"]
            start_position => "beginning"
            sincedb_path => "/dev/null"
            codec => multiline {
            pattern => "^%{TIMESTAMP_ISO8601} "
            negate => true
            what => previous
            }
      }

      file {
            type => "error"
            path => ["/var/log/tomcat8/**/*-error.log"]
            start_position => "beginning"
            sincedb_path => "/dev/null"
            codec => multiline {
            pattern => "^%{TIMESTAMP_ISO8601} "
            negate => true
            what => previous
            }
      }
      file {
            type => "location"
            path => ["/var/log/tomcat8/**/*-location.log"]
            start_position => "beginning"
            sincedb_path => "/dev/null"
            codec => multiline {
            pattern => "^%{TIMESTAMP_ISO8601} "
            negate => true
            what => previous
            }
  }
  filter {
  
          if [type] == "audit" {
              grok {
              match => [ "message",
                      "%{TIMESTAMP_ISO8601:timestamp_match} (\[%{DATA:thread}\])?( )?%{LOGLEVEL:level}%{SPACE}%{USERNAME:zhost}%{SPACE}%{JAVAFILE:javaClass} %{USERNAME:orgId} %{USERNAME:loginUserId} %{USERNAME:sessionId} %{USERNAME:txnId} (?<email>[\w.+=:-]+@[0-9A-Za-z][0-9A-Za-z-]{0,62}(?:[.](?:[0-9A-Za-z][0-9A-Za-z‌​-]{0,62}))*) %{URI:url}"]
              }
          }
          else if [type] == "app" {
              grok {
              match => [ "message",
                      "%{TIMESTAMP_ISO8601:timestamp_match} (\[%{DATA:thread}\])?( )?%{LOGLEVEL:level}%{SPACE}%{USERNAME:zhost} %{JAVAFILE:javaClass}%{GREEDYDATA:infoMsg}"]
              }
          }
  
          else if [type] == "error" {
              grok {
              match => [ "message",
                      "%{TIMESTAMP_ISO8601:timestamp_match} (\[%{DATA:thread}\])?( )?%{LOGLEVEL:level}%{SPACE}%{USERNAME:zhost} %{JAVAFILE:javaClass}%{SPACE}%{USERNAME:orgId}%{GREEDYDATA:errorMessage}"]
              }
          }
          else if [type] == "location" {
              grok {
              match => [ "message","%{TIMESTAMP_ISO8601:timestamp_match} (\[%{DATA:thread}\])?( )?%{LOGLEVEL:level}%{SPACE}%{USERNAME:zhost} %{JAVAFILE:javaClass}%{SPACE}%{USERNAME:orgId}%{SPACE}%{USERNAME:userId}%{SPACE}%{TIMESTAMP_ISO8601:lastSeenTime}%{SPACE}%{USERNAME:longitude}%{SPACE}%{USERNAME:latitude}"]
          }
  		}


  output {
      elasticsearch {
          hosts => ["172.31.8.70:9200"]
          manage_template => false
          index => "logs"
      }
  }

Below exception it is throwing, couldn't able to find out what is wrong with this

Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"=>\" at line 50, column 12 (byte 1339) after input {\n file {\n type => \"audit\"\n path => [\"/var/log/tomcat8/**/*-audit.log\"]\n start_position => \"beginning\"\n sincedb_path => \"/tmp/sincedb_file\"\n codec => multiline {\n pattern => \"^%{TIMESTAMP_ISO8601} \"\n negate => true\n what => previous\n }\n }\n\n file {\n type => \"app\"\n path => [\"/var/log/tomcat8/**/*-info.log\"]\n start_position => \"beginning\"\n sincedb_path => \"/tmp/sincedb_app_file\"\n codec => multiline {\n pattern => \"^%{TIMESTAMP_ISO8601} \"\n negate => true\n what => previous\n }\n }\n\n file {\n type => \"error\"\n path => [\"/var/log/tomcat8/**/*-error.log\"]\n start_position => \"beginning\"\n sincedb_path => \"/tmp/sincedb_err_file\"\n codec => multiline {\n pattern => \"^%{TIMESTAMP_ISO8601} \"\n negate => true\n what => previous\n }\n }\n file {\n type => \"location\"\n path => [\"/var/log/tomcat8/**/*-location.log\"]\n start_position => \"beginning\"\n sincedb_path => \"/tmp/sincedb_loc_file\"\n codec => multiline {\n pattern => \"^%{TIMESTAMP_ISO8601} \"\n negate => true\n what => previous\n }\n}\nfilter {\n\n if ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in block in compile_sources'", "org/jruby/RubyArray.java:2584:in map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:156:in initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:27:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:326:in block in converge_state'"]}

2

Looks like your input section is missing a closing } before the filter section. You might want to check your indentation of the brackets on your your file input there and your multiline codec.

1 Like
3

@jong99 thank you for the reply. Yes i missed } at line number 50. Fixed my issue.

Below command also tell me what wrong with my logstash

/usr/share/logstash/bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/logstash.conf

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.