LogStash::ConfigurationError but ConfigurationOK - Logstash 2.4.0


(Carl) #1

I am tring to upgrade from Logstash 1.4.4 to Logstash 2.4.0

I have reused the same configuration file that we used before for parsing our application logs

I am running Logstash 2.4.0 in debug mode and getting that block that does not indicate that much

{:timestamp=>"2016-11-10T15:49:43.008000-0500", :message=>"Pipeline aborted due to error", :exception=>"LogStash::ConfigurationError", :backtrace=>["/cceapps/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/config/mixin.rb:88:in `config_init'", "org/jruby/RubyHash.java:1342:in `each'", "/cceapps/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/config/mixin.rb:72:in `config_init'", "/cceapps/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/outputs/base.rb:79:in `initialize'", "/cceapps/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/output_delegator.rb:74:in `register'", "/cceapps/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:181:in `start_workers'", "org/jruby/RubyArray.java:1613:in `each'", "/cceapps/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:181:in `start_workers'", "/cceapps/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:136:in `run'", "/cceapps/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/agent.rb:491:in `start_pipeline'"], :level=>:error, :file=>"logstash/agent.rb", :line=>"493", :method=>"start_pipeline"}
{:timestamp=>"2016-11-10T15:49:46.016000-0500", :message=>"stopping pipeline", :id=>"main", :file=>"logstash/agent.rb", :line=>"406", :method=>"shutdown_pipelines"}
{:timestamp=>"2016-11-10T15:49:46.019000-0500", :message=>"Closing inputs", :level=>:info, :file=>"logstash/pipeline.rb", :line=>"384", :method=>"shutdown"}
{:timestamp=>"2016-11-10T15:49:46.021000-0500", :message=>"stopping", :plugin=>"LogStash::Inputs::File", :level=>:debug, :file=>"logstash/inputs/base.rb", :line=>"81", :method=>"do_stop"}
{:timestamp=>"2016-11-10T15:49:46.023000-0500", :message=>"Closed inputs", :level=>:info, :file=>"logstash/pipeline.rb", :line=>"386", :method=>"shutdown"}

When i run the configtest parameter i get a Coonfiguration OK message

$ ./bin/logstash --configtest --config ./conf/app.conf
Configuration OK

My configuration file uses custom patterns created in files that exist in patterns_dir

Here is my configuration file

input {
file {
path => "/apps/wlserver/logs/pce.log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}

filter {
multiline {
# Lines starting with a date are new entries (ie. not multiline).
pattern => "^%{MYTIMESTAMP} "
negate => true
what => "previous"
}

if "multiline" in [tags] {
    # Multiline messages contain an fullqualifiedname and stacktrace.
    grok {
      patterns_dir => "./patterns"
      match => [ "message", "^%{MYTIMESTAMP:datetime}\s+\[%{GREEDYDATA:brackettext}\]\s+\{%{LOGLEVEL:severity}\}\s+%{GREEDYDATA:jobname}\s+\(%{JAVACLASS:javaclass}(:(?<linenumber>\d+))?\)\s+-\s+%{GREEDYDATA:messagetext}\n(?m)(?<stacktrace>(?<fullqualifiedname>([a-zA-Z0-9-]+\.)*(?<classname>[A-Za-z0-9$_]+))?%{GREEDYDATA})" ]
    }
} else {
    # Single line messages don't contain an classname or stacktrace. Add empty fields.
    grok {
        patterns_dir => "./patterns"
        match => [ "message", "^%{MYTIMESTAMP:datetime}\s+\[%{GREEDYDATA:brackettext}\]\s+\{%{LOGLEVEL:severity}\}\s+%{GREEDYDATA:jobname}\s+\(%{JAVACLASS:javaclass}(:(?<linenumber>\d+))?\)\s+-\s+%{GREEDYDATA:messagetext}$" ]
        add_field => {
            "fullqualifiedname" => ""
            "classname" => ""
            "stacktrace" => ""
        }
    }
}


# Drop entries based on severity
if [severity] != "ERROR" and [severity] != "WARN" {
    drop { }

}

}

output {
stdout { codec => rubydebug }
elasticsearch {
cluster => "myculster"
flush_size => 500
host => "hostname"
idle_flush_time => 30
index => "logstash-%{+YYYY-MM-dd}"
node_name => "dev"
port => "19001"
protocol => "http"
workers => 1
}
}

sorry for the formatting i could not do better than that

Regards


(Magnus B├Ąck) #2

I am running Logstash 2.4.0 in debug mode and getting that block that does not indicate that much

Yeah, it's a bug that the real error message is lost.

You're using a bunch of options to the elasticsearch output that were removed in Logstash 2.0. Please study the documentation and remove those that no longer apply. I'd expect these backwards incompatible changes should be described in the 2.0 release notes.


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.