Logstash and MongoDB input

Elastic Stack Logstash
1

Hi!
Logstash version: 7.17.0
Could you help me with logstash-mongodb issue, in view of input plugin for Logstash.
I work on some task, where I need to get logs from postgre and mongo.
Regarding to postgre - everything is fine.
But with mongo I faced with lot of issues.
First i tried to work with logstash-input-mongodb, probably it was great solution.
Unfortunately there is no support for many years, and it's difficult to use it with current versions of software.

https://github.com/phutchins/logstash-input-mongodb

Second way - jdbc input.
I tried different drivers.
mongodb-jdbc-1.0.3.jar
mongojdbc1.6.jar
mongojdbc4.0.jar

But still got NoClassDefFoundError.

Please, help me to fix it or probably there is another solution to send collection data from mongodb to Elasticsearch.

conf.d/mongo.conf

input {
        jdbc {
                jdbc_driver_library => "/etc/logstash/mongojdbc4.0.jar"
                jdbc_driver_class => "com.dbschema.MongoJdbcDriver"
                jdbc_connection_string => "jdbc:mongodb://username:password@127.0.0.1:27017/mydb"
                jdbc_user => "username"
                jdbc_password => "password"
                statement => "db.final_fields.find({},{'_id': false});"
  }
}

output {
        stdout {
                codec => rubydebug
        }
}

sudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/mongo.conf

Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2022-02-18 00:14:36.580 [main] runner - Starting Logstash {"logstash.version"=>"7.17.0", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
[INFO ] 2022-02-18 00:14:36.584 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true]
[WARN ] 2022-02-18 00:14:36.761 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2022-02-18 00:14:37.482 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[INFO ] 2022-02-18 00:14:38.335 [Converge PipelineAction::Create<main>] Reflections - Reflections took 50 ms to scan 1 urls, producing 119 keys and 417 values 
[WARN ] 2022-02-18 00:14:38.889 [Converge PipelineAction::Create<main>] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-02-18 00:14:38.911 [Converge PipelineAction::Create<main>] jdbc - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2022-02-18 00:14:39.080 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/mongo.conf"], :thread=>"#<Thread:0x486c78d2 run>"}
[INFO ] 2022-02-18 00:14:39.572 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>0.49}
[INFO ] 2022-02-18 00:14:39.635 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2022-02-18 00:14:39.674 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[ERROR] 2022-02-18 00:14:40.013 [[main]<jdbc] jdbc - Unable to connect to database. Tried 1 times {:message=>"Java::JavaLang::NoClassDefFoundError: com/mongodb/MongoClientURI", :exception=>Sequel::DatabaseConnectionError, :cause=>java.lang.NoClassDefFoundError: com/mongodb/MongoClientURI, :backtrace=>["com.dbschema.mongo.JMongoClient.<init>(com/dbschema/mongo/JMongoClient.java:27)", "com.dbschema.mongo.MongoService.<init>(com/dbschema/mongo/MongoService.java:28)", "com.dbschema.MongoJdbcDriver.connect(com/dbschema/MongoJdbcDriver.java:56)", "java.sql.DriverManager.getConnection(java/sql/DriverManager.java:677)", "java.sql.DriverManager.getConnection(java/sql/DriverManager.java:251)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:62)", "jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:566)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:426)", "org.jruby.javasupport.JavaMethod.invokeStaticDirect(org/jruby/javasupport/JavaMethod.java:358)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_52_dot_0.lib.sequel.adapters.jdbc.connect(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.52.0/lib/sequel/adapters/jdbc.rb:225)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_52_dot_0.lib.sequel.connection_pool.make_new(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.52.0/lib/sequel/connection_pool.rb:122)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_52_dot_0.lib.sequel.connection_pool.threaded.assign_connection(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.52.0/lib/sequel/connection_pool/threaded.rb:209)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_52_dot_0.lib.sequel.connection_pool.threaded.acquire(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.52.0/lib/sequel/connection_pool/threaded.rb:139)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_52_dot_0.lib.sequel.connection_pool.threaded.hold(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.52.0/lib/sequel/connection_pool/threaded.rb:91)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_52_dot_0.lib.sequel.database.connecting.synchronize(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.52.0/lib/sequel/database/connecting.rb:269)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_52_dot_0.lib.sequel.database.connecting.test_connection(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.52.0/lib/sequel/database/connecting.rb:278)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_52_dot_0.lib.sequel.database.misc.initialize(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.52.0/lib/sequel/database/misc.rb:175)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_52_dot_0.lib.sequel.database.connecting.connect(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.52.0/lib/sequel/database/connecting.rb:57)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_52_dot_0.lib.sequel.core.connect(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.52.0/lib/sequel/core.rb:124)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_2_dot_2.lib.logstash.plugin_mixins.jdbc.jdbc.jdbc_connect(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.2.2/lib/logstash/plugin_mixins/jdbc/jdbc.rb:117)", "org.jruby.RubyKernel.loop(org/jruby/RubyKernel.java:1442)", "org.jruby.RubyKernel$INVOKER$s$0$0$loop.call(org/jruby/RubyKernel$INVOKER$s$0$0$loop.gen)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_2_dot_2.lib.logstash.plugin_mixins.jdbc.jdbc.jdbc_connect(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.2.2/lib/logstash/plugin_mixins/jdbc/jdbc.rb:114)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_2_dot_2.lib.logstash.plugin_mixins.jdbc.jdbc.open_jdbc_connection(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.2.2/lib/logstash/plugin_mixins/jdbc/jdbc.rb:157)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_2_dot_2.lib.logstash.plugin_mixins.jdbc.jdbc.execute_statement(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.2.2/lib/logstash/plugin_mixins/jdbc/jdbc.rb:214)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_2_dot_2.lib.logstash.inputs.jdbc.execute_query(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.2.2/lib/logstash/inputs/jdbc.rb:339)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_2_dot_2.lib.logstash.inputs.jdbc.run(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.2.2/lib/logstash/inputs/jdbc.rb:307)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.inputworker(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:409)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_input(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:400)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:318)", "java.lang.Thread.run(java/lang/Thread.java:829)"]}
[WARN ] 2022-02-18 00:14:40.017 [[main]<jdbc] jdbc - Exception when executing JDBC query {:exception=>Sequel::DatabaseConnectionError, :message=>"Java::JavaLang::NoClassDefFoundError: com/mongodb/MongoClientURI", :cause=>"java.lang.NoClassDefFoundError: com/mongodb/MongoClientURI"}
[INFO ] 2022-02-18 00:14:40.159 [[main]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>"main"}
[INFO ] 2022-02-18 00:14:40.220 [Converge PipelineAction::Delete<main>] pipelinesregistry - Removed pipeline from registry successfully {:pipeline_id=>:main}
[INFO ] 2022-02-18 00:14:40.253 [LogStash::Runner] runner - Logstash shut down.
2

Though I'm not familiar with MongoDB, last resort could be using Exec input plugin with creating some bash script which retrieve data from MongoDB.

Bash script could be something like:

 #!/bin/bash
 mongo --quiet << 'EOT'
 use db01;
 db.coll_01.                             // collection 選択
    find({type:'A'},{_id:0}).            // typeが'A'のレコード
    sort({_id:-1}).                      // _idで逆向きにソート
    limit(3).                            // 個数制限
    toArray().reverse().                 // 反転したリスト取得
    forEach(S=>print(JSON.stringify(S)));// 要素表示
 EOT

The page just came up from a quick Google search. It is in Japanese and please use some translator if you want to read.

3

Try putting you .jar in the logstash-core library folder. Having it in other directories in the past cause this type of issues due to permissions I believe. Will be something like this depending on your version and OS. Then change your reference to that path and try again. If that doesn't work try completely removing the jdbc_driver_library setting from your input with the jar in the same folder as above.

/usr/share/logstash/logstash-core/lib/jars/

4

Thank you, problem was with incorrect jdbc driver.
Currently i should configure the same function on logstash 2.4.1 version
And as I understand, there was another path not /usr/share/logstash/logstash-core/lib/jars/
Do you know path of driver libraries in old versions?

5

Thank you, for alternative resolution

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.