Logstash 6.1.1 : jar-dependencies conflicts

Elastic Stack Logstash
1

Hello !

I'm running Logstash 6.1.1
I've updated a logstash plugin : logstash-input-udp.

Since this update I'm having apparently a conflict between different versions of jar-dependencies.

Here's what I find in the logs :

Gem::LoadError: You have already activated jar-dependencies 0.3.12, but your Gemfile requires jar-dependencies 0.3.11. Prepending bundle exec to your command may solve this.

I did some actions, but neither succeeded :

  1. Removed the logstash-input-udp plugin

  2. Re-installed the plugin and updated it

  3. Removed the whole /usr/share/logstash tree and the logstash rpm. Then re-installed Logstash 6.1.1

I always have the same error.
But when re-installing Logstash, instead of having these errors in /var/log/messages ; I'm having now this error in the logstash logs directly :

ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Something is wrong with your configuration.", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:89:in config_init'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:128:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:23:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/plugins/plugin_factory.rb:88:inplugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:114:in plugin'", "(eval):92:in'", "org/jruby/RubyKernel.java:994:in eval'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:86:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:171:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:335:in block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:332:in block in converge_state'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:319:in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:inblock in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:inconverge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:343:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

I'm not a java nor a ruby guru and I'm rather stucked at this point.

Thank you in advance for your help.

2

Oy. That's not a very helpful error message :weary:

  1. what is the pipeline configuration (or at least, what inputs, outputs, and filters does it use)?
  2. what plugins are currently installed? bin/logstash-plugin list --verbose
3

Hello,

Sorry for the late reply: I was on holiday.

Here are the details you asked for :

  • plugins list :
    =============

./bin/logstash-plugin list --verbose
logstash-codec-cef (5.0.2)
logstash-codec-collectd (3.0.8)
logstash-codec-dots (3.0.6)
logstash-codec-edn (3.0.6)
logstash-codec-edn_lines (3.0.6)
logstash-codec-es_bulk (3.0.6)
logstash-codec-fluent (3.1.5)
logstash-codec-graphite (3.0.5)
logstash-codec-json (3.0.5)
logstash-codec-json_lines (3.0.5)
logstash-codec-line (3.0.6)
logstash-codec-msgpack (3.0.7)
logstash-codec-multiline (3.0.8)
logstash-codec-netflow (3.9.0)
logstash-codec-plain (3.0.6)
logstash-codec-rubydebug (3.0.5)
logstash-filter-aggregate (2.7.2)
logstash-filter-anonymize (3.0.6)
logstash-filter-cidr (3.1.2)
logstash-filter-clone (3.0.5)
logstash-filter-csv (3.0.7)
logstash-filter-date (3.1.9)
logstash-filter-de_dot (1.0.3)
logstash-filter-dissect (1.1.2)
logstash-filter-dns (3.0.7)
logstash-filter-drop (3.0.5)
logstash-filter-elasticsearch (3.2.1)
logstash-filter-fingerprint (3.1.2)
logstash-filter-geoip (5.0.2)
logstash-filter-grok (4.0.0)
logstash-filter-jdbc_streaming (1.0.3)
logstash-filter-json (3.0.5)
logstash-filter-kv (4.0.3)
logstash-filter-metrics (4.0.5)
logstash-filter-mutate (3.2.0)
logstash-filter-ruby (3.1.3)
logstash-filter-sleep (3.0.6)
logstash-filter-split (3.1.6)
logstash-filter-syslog_pri (3.0.5)
logstash-filter-throttle (4.0.4)
logstash-filter-translate (3.0.4)
logstash-filter-truncate (1.0.4)
logstash-filter-urldecode (3.0.6)
logstash-filter-useragent (3.2.2)
logstash-filter-xml (4.0.5)
logstash-input-beats (5.0.4)
logstash-input-dead_letter_queue (1.1.2)
logstash-input-elasticsearch (4.1.1)
logstash-input-exec (3.1.5)
logstash-input-file (4.0.3)
logstash-input-ganglia (3.1.3)
logstash-input-gelf (3.0.7)
logstash-input-generator (3.0.5)
logstash-input-graphite (3.0.4)
logstash-input-heartbeat (3.0.5)
logstash-input-http (3.0.8)
logstash-input-http_poller (4.0.4)
logstash-input-imap (3.0.5)
logstash-input-jdbc (4.3.2)
logstash-input-kafka (8.0.2)
logstash-input-pipe (3.0.6)
logstash-input-rabbitmq (6.0.2)
logstash-input-redis (3.1.6)
logstash-input-s3 (3.1.8)
logstash-input-snmptrap (3.0.5)
logstash-input-sqs (3.0.6)
logstash-input-stdin (3.2.5)
logstash-input-syslog (3.2.4)
logstash-input-tcp (5.0.2)
logstash-input-twitter (3.0.7)
logstash-input-udp (3.3.0)
logstash-input-unix (3.0.6)
logstash-output-cloudwatch (3.0.7)
logstash-output-csv (3.0.6)
logstash-output-elasticsearch (9.0.2)
logstash-output-email (4.0.6)
logstash-output-file (4.2.1)
logstash-output-graphite (3.1.4)
logstash-output-http (5.1.1)
logstash-output-kafka (7.0.4)
logstash-output-lumberjack (3.1.5)
logstash-output-nagios (3.0.5)
logstash-output-null (3.0.4)
logstash-output-pagerduty (3.0.6)
logstash-output-pipe (3.0.5)
logstash-output-rabbitmq (5.0.3)
logstash-output-redis (4.0.3)
logstash-output-s3 (4.0.13)
logstash-output-sns (4.0.6)
logstash-output-sqs (5.0.2)
logstash-output-stdout (3.1.3)
logstash-output-tcp (5.0.2)
logstash-output-udp (3.0.5)
logstash-output-webhdfs (3.0.5)
logstash-patterns-core (4.1.2)

  • Pipeline configuration
    ====================

  1. INPUTS :

    • file
    • udp / codec => netflow (*)

  2. OUTPUTS : elasticsearch

  3. FILTERS :

    • mutate
    • grok
    • geoip
    • date
    • ruby

(*) As I mentionned it, it's after the logstash-input-udp plugin update, that the mess appeared.
But even if I remove this input from my configuration, logstash does not run.

Thank you in advance for your help !

4

did you use the bin/logstash-plugin update logstash-input-udp command to update, or some other means? The above is supposed to resolve these kinds of conflicts for us...

5

Hi Ry,

Here's what I did exactly :

  • I first tried to update it through the logstash-plugin update command as you said above.
    However I couldn't do it (and I don't remember the error message). But I know I had to update my system (sudo yum update).

  • As the problem didn't disappear, even when removing the config file using input-udp, I removed the the plugin by using

./bin/logstash-plugin remove logstash-input-udp

Then I re-installed it :

./bin/logstash-plugin install logstash-input-udp

Still, the problem didn't disappear.

So, I deinstalled all logstash 6.1.1 and re-installed it completely :

  • sudo rpm -e logstash-6.1.1-1.noarch
  • sudo rpm -ihv logstash-6.1.1.rpm

But it hasn't change the situation.

I don't know what to do. And I'm ready to de-install the whole ELK 6.1.1 stack and install the last stable version of the stack.

Thanks once again.

6
  • On a fresh install of logstash via rpm (without trying the plugin update), are you getting the above issues?
  • There are a couple patch releases of Logstash since 6.1.1 that are fully compatible with any 6.1.x version of the rest of the Elastic Stack -- 6.1.4 is the most recent, and it includes a bump of logstash-input-udp from v3.1.3 to v3.2.1, so you may not have to do the plugin upgrade on your own.

For good measure, what JVM/JDK are you using?

  • java -version

  • echo ${JAVA_HOME?}
7
  • Yes. I reinstalled Logstash 6.1.1 via rpm and the issues are still there.
  • I didn't apply any patches after the reinstall. I will do it on monday.
  • I'm using the OpenJDK 1.8 on a RedHat Linux Server 7.4

===> java -version gives me this :

 openjdk version "1.8.0_161"
OpenJDK Runtime Environment (build 1.8.0_161-b14)
OpenJDK 64-Bit Server VM (build 25.161-b14, mixed mode)

The error message makes me think that on updating the udp plugin I changed something in java itself (?). But as I said it above, I'm not a java person.
Never the less, I'll install the logstash patches, and I'll see what happens.

8

Hi Ry,

I upgraded logstash from 6.1.1 to 6.1.4 (the last version for 6.1).
It didn't help. I still have the same error log (see below).

As I told you in my last message, the issue seems not to be in the core of logstash but related to the java/Gem/ruby stuff :

before, I re-install logstash the error message was the following :

Gem::LoadError: You have already activated jar-dependencies 0.3.12, but your Gemfile requires jar-dependencies 0.3.11. Prepending bundle exec to your command may solve this.

And I don't know how to go back to jar-dependencies 0.3.11, if this ever makes sense.

When searching for the jar-dependencies.gemspec file, I find this :

find . -name jar-dependencies.gemspec -print
./vendor/bundle/jruby/2.3.0/gems/jar-dependencies-0.3.12/jar-dependencies.gemspec

Logstash log error :

Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Something is wrong with your configuration.", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:89:in config_init'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:128:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:23:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/plugins/plugin_factory.rb:88:inplugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:114:in plugin'", "(eval):92:in'", "org/jruby/RubyKernel.java:994:in eval'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:86:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:171:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:335:in block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:332:in block in converge_state'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:319:in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:inblock in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:inconverge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:343:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

9

Finally, it worked !

First, I recognize that I've missed another error message in the logstash logs :frowning:
I use geo-location and after the first reinstall : my geo database has been removed.
I reinstalled my geo database.

Moreover, I removed the whole subdir :
logstash/vendor/bundle/jruby/2.3.0/gems/jar-dependencies-0.3.12 (which was present even though I re-installed logstash multiple times)

  • removed the logstash package (yum remove)
  • re-installed logstash-6.1.1 (yum install)
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.