Logstash Google Pub Sub - results in error java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String

Elastic Stack Logstash
1

I installed logstash on a debian OS and then installed the Google Pub Sub plugin . Simple pipeline test works fine, but using Google pub/sub as input results in below error.

last line of stacktrace:
[ERROR] 2018-05-15 20:48:06.544 [LogStash::Runner] Logstash - java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;J)V

sudo bin/logstash --version
logstash 6.2.4

/usr/share/logstash$ sudo bin/logstash -f gcp-pubsub-logstash.conf --config.reload.automatic
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2018-05-15 20:48:02.415 [main] scaffold - Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[INFO ] 2018-05-15 20:48:02.443 [main] scaffold - Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[WARN ] 2018-05-15 20:48:03.490 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2018-05-15 20:48:03.824 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.2.4"}
[INFO ] 2018-05-15 20:48:04.243 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2018-05-15 20:48:05.908 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[INFO ] 2018-05-15 20:48:06.085 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x2ef216d6@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 run>"}
[INFO ] 2018-05-15 20:48:06.140 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Pipelines running {:count=>1, :pipelines=>["main"]}
[INFO ] 2018-05-15 20:48:06.515 [[main]-pipeline-manager] pipeline - Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x2ef216d6@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 run>"}
[ERROR] 2018-05-15 20:48:06.544 [LogStash::Runner] Logstash - java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;J)V

cat gcp-pubsub-logstash.conf

The # character at the beginning of a line indicates a comment. Use

comments to describe your configuration.

input {
google_pubsub {
# Your GCP project id (name)
project_id => "pinkesh-autocomplete-202716"

    # The topic name below is currently hard-coded in the plugin. You
    # must first create this topic by hand and ensure you are exporting
    # logging to this pubsub topic.
    topic => "product-ingestion-topic"

    # The subscription name is customizeable. The plugin will attempt to
    # create the subscription (but use the hard-coded topic name above).
    subscription => "mysubs"

    # If you are running logstash within GCE, it will use
    # Application Default Credentials and use GCE's metadata
    # service to fetch tokens.  However, if you are running logstash
    # outside of GCE, you will need to specify the service account's
    # JSON key file below.
    #json_key_file => "/home/erjohnso/pkey.json"
}

}
filter {
mutate {
add_field => { "messageId" => "%{[@metadata][pubsub_message][messageId]}" }
}
}
output { stdout { codec => rubydebug } }

/usr/share/logstash$ java -version
openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-1~deb9u1-b11)
OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode)

I had to create a symlink for Logstash to install correctly since it is by default using /usr/bin/java

ls -l /usr/bin/java
lrwxrwxrwx 1 root root 42 May 15 19:58 /usr/bin/java -> /usr/lib/jvm/java-8-openjdk-amd64/bin/java

echo $JAVA_HOME
/usr/lib/jvm/java-8-openjdk-amd64

echo $PATH
/usr/lib/jvm/java-8-openjdk-amd64/bin:/opt/bitnami/elasticsearch/bin:/opt/bitnami/apache2/bin:/opt/bitnami/common/bin:/opt/bitnami/java/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games

2

It looks like Logstash reverted their guava version between 6.2.3 to 6.2.4. Here's the commit that I think did it: https://github.com/elastic/logstash/commit/a425a422e03087ac34ad6949f7c95ec6d27faf14

The 1.1 version imports guava 19 and the 1.5 version imports Guava 22. It looks like google-java-format imports it was reverted because it imports javac-shaded which is GPL licensed.

Here's the JARs I find in the docker images:

user@elasticstack:~$ docker run -it docker.elastic.co/logstash/logstash:6.2.3 bash
WARNING: IPv4 forwarding is disabled. Networking will not work.
bash-4.2$ find . | grep guava
./logstash-core/lib/jars/guava-22.0.jar
bash-4.2$ 

user@elasticstack:~$ docker run -it docker.elastic.co/logstash/logstash:6.2.4 bash
WARNING: IPv4 forwarding is disabled. Networking will not work.
bash-4.2$ find . | grep guava
./logstash-core/lib/jars/guava-19.0.jar
bash-4.2$

We have an open issue in the Logstash plugin repository and a PR.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.