Gelf in Kafka out for logstash

Elastic Stack Logstash
1

Im trying to use Logstash with a GELF input and outputting to Kafka and not getting any logs.

My config file looks like this -

input {
   udp {
     host => "ip of loadbalancer"
     port => 9900
     type => gelf
     }
}


filter {
   date {
       match => ["timestamp", "MMM dd HH:mm:ss" ]
       target => "timestamp"
       timzone => "Europe/London"
   }
}


output {
   kafka {
        host => "ip of kafka server"
        port => 9092
        type => kafka
   }
}

When running im getting the errors -

[2020-06-23T15:40:32,101][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"if\", [A-Za-z0-9_-], '\"', \"'\", \"}\" at line 26, column 1 (byte 339) after output {\n   kafka {\n        host => [\"52.1x.x.x\"]\n        port => 9092\n        type => kafka\n   }\n\n", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:58:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:66:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:28:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:27:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:181:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:67:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:43:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:342:in `block in converge_state'"]}
[2020-06-23T15:40:36,632][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2020-06-23T15:40:39,065][INFO ][logstash.runner          ] Logstash shut down.

and - [2020-06-23T15:21:39,299][ERROR][logstash.javapipeline    ][main] Worker loop initialization error {:pipeline_id=>"main", :error=>"Missing Filter End Vertex", :exception=>Java::JavaLang::IllegalStateException, :stacktrace=>"org.logstash.config.ir.CompiledPipeline$CompiledExecution.lambda$compileFilters$1(org/logstash/config/ir/CompiledPipeline.java:386)\njava.util.Optional.orElseThrow(java/util/Optional.java:408)\norg.logstash.config.ir.CompiledPipeline$CompiledExecution.compileFilters(org/logstash/config/ir/CompiledPipeline.java:386)\norg.logstash.config.ir.CompiledPipeline$CompiledExecution.<init>(org/logstash/config/ir/CompiledPipeline.java:369)\norg.logstash.config.ir.CompiledPipeline$CompiledUnorderedExecution.<init>(org/logstash/config/ir/CompiledPipeline.java:329)\norg.logstash.config.ir.CompiledPipeline.buildExecution(org/logstash/config/ir/CompiledPipeline.java:153)\norg.logstash.execution.WorkerLoop.<init>(org/logstash/execution/WorkerLoop.java:63)\njdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\njdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(jdk/internal/reflect/NativeConstructorAccessorImpl.java:62)\njdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(jdk/internal/reflect/DelegatingConstructorAccessorImpl.java:45)\njava.lang.reflect.Constructor.newInstance(java/lang/reflect/Constructor.java:490)\norg.jruby.javasupport.JavaConstructor.newInstanceDirect(org/jruby/javasupport/JavaConstructor.java:253)\norg.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:939)\norg.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)\nusr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.init_worker_loop(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:499)\nusr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$init_worker_loop$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb)\nusr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:267)\norg.jruby.RubyProc.call(org/jruby/RubyProc.java:318)\njava.lang.Thread.run(java/lang/Thread.java:834)", :thread=>"#<Thread:0x756fda0c sleep>"}

and

[2020-06-23T15:21:39,358][ERROR][logstash.javapipeline    ][main] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<RuntimeError: Some worker(s) were not correctly initialized>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:270:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:170:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:125:in `block in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/logstash.conf"], :thread=>"#<Thread:0x756fda0c run>"}

Can anyone help?

2

Could be missing a closing } for the output section.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.