Logstash failing after upgrade into the version 2.x

Elastic Stack Logstash
1

hi All, I recently upgraded logstash as per https://www.elastic.co/guide/en/logstash/current/installing-logstash.html#package-repositories and I'm unable to start the logstash service which is now failing with below error logged in logs:

{:timestamp=>"2016-09-01T11:46:56.873000+0100", :message=>"Pipeline aborted due to error", :exception=>"LogStash::ConfigurationError", :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/config/mixin.rb:88:in `config_init'",

Any help will be much appreciated, cheers, Tomek

2

Is that really the only error message? What if you run logstash --configtest on your configuration files?

3

hi Magnus, thanks for the prompt response logstash --configtest results in the "Configuration OK", see the contents of my filter folders:

ls -l /etc/logstash/conf.d/
total 20
-rw-r--r-- 1 kibana4 kibana4 438 Sep 1 11:42 01-inputs.conf
-rw-rw-r-- 1 kibana4 kibana4 1195 Jul 12 07:09 10-syslog.conf
-rw-rw-r-- 1 kibana4 kibana4 1302 Jul 12 07:05 11-pfsense.conf
-rw-rw-r-- 1 kibana4 kibana4 127 Jul 12 07:03 30-outputs.conf
drwxr-xr-x 2 kibana4 kibana4 4096 Jul 12 07:10 patterns

Shall i try to run the configtest against those files or does the configtest goes through them automatically?

4

It does not use those files automatically. Run logstash --configtest -f /etc/logstash/conf.d.

I can tell you already that it's complaining about the patterns file which, I assume, is a grok pattern file. Logstash reads all files in the configuration directory it's pointed to.

5

thanks Magnus, it does not show me any output regarding the grok patterns- do i need to update them separately?

6

You probably don't need to update them at all, but they can't be stored in /etc/logstash/conf.d since Logstash reads all files in that directory as configuration files.

7

what will be your suggestion to move the patterns folder to?

8

/etc/logstash for example.

9

Still the same error (with the grok patterns in /etc/logstash). I'm pasting the error message from logstash.log:

{:timestamp=>"2016-09-01T16:18:58.186000+0100", :message=>"Pipeline aborted due to error", :exception=>"LogStash::ConfigurationError", :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/config/mixin.rb:88:in config_init'", "org/jruby/RubyHash.java:1342:ineach'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/config/mixin.rb:72:in config_init'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/outputs/base.rb:79:ininitialize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/output_delegator.rb:74:in register'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:181:instart_workers'", "org/jruby/RubyArray.java:1613:in each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:181:instart_workers'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:136:in run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/agent.rb:491:instart_pipeline'"], :level=>:error}
{:timestamp=>"2016-09-01T16:19:01.192000+0100", :message=>"stopping pipeline", :id=>"main"}

10

Is that really the only message in the log? The stacktrace indicates that you're using an obsolete configuration option for some plugin, but I'd expect there to be a message indicating which option it's complaining about.

11

That is the only message logged into the log files- will you be able to point me into where can I look for the obsolete option. Let me know if you require any additional logs (elasticsearch, syslog)

12

The documentation for the plugins you use should state if the option is obsolete (or if it's missing from the documentation; that would obviously be a clear indication), but if you post your configuration we might be able to spot the problem right away.

13

Thanks Magnus and apologies for the delay in replying i will post the conf files shortly. Just to double check with you as plugin configuration you are referring to the contents of /etc/logstash/conf.d/

To let you know i dont have any additional plugins for elasticsearch. I had kopf and bigdesk, but removed them prior to upgrade.

Cheers, Tomek

14

Just to double check with you as plugin configuration you are referring to the contents of /etc/logstash/conf.d/

Yes.

15

OK, see the files in my owncloud drive: http://yourls.dyndns.biz/logstash

I've included below as a single text file:

ls -l /etc/logstash/conf.d/
-rw-r--r-- 1 kibana4 kibana4 438 Sep 1 11:42 01-inputs.conf
-rw-rw-r-- 1 kibana4 kibana4 1195 Jul 12 07:09 10-syslog.conf
-rw-rw-r-- 1 kibana4 kibana4 1295 Sep 1 16:18 11-pfsense.conf
-rw-rw-r-- 1 kibana4 kibana4 125 Sep 2 11:57 30-outputs.conf

16

In your elasticsearch output configuration, change

host => localhost

to

hosts => ["localhost"]

This is documented in the 2.0 breaking changes document.

17

Magnus, Fantastic! worked as a charm and many, many thanks as it was causing a massive pain for me to sort it.
I can confirm that all is working fine now and both logstash and elasticsearch are working fine

18

I've tried the configuration above but I'm still getting this error:

{:timestamp=>"2016-09-21T14:13:43.656000+0200", :message=>"Pipeline aborted due to error", :exception=>"Grok::PatternError", :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.3/lib/grok-pure.rb:123:in compile'", "org/jruby/RubyKernel.java:1479:inloop'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.3/lib/grok-pure.rb:93:in compile'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:264:inregister'", "org/jruby/RubyArray.java:1613:in each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:259:inregister'", "org/jruby/RubyHash.java:1342:in each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:255:inregister'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:182:in start_workers'", "org/jruby/RubyArray.java:1613:ineach'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:182:in start_workers'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:136:inrun'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/agent.rb:491:in `start_pipeline'"], :level=>:error}
{:timestamp=>"2016-09-21T14:13:46.664000+0200", :message=>"stopping pipeline", :id=>"main"}

Any suggestions?

19

@Soren, please start a new thread for your question.