Logstash and MongoDB input

serfisar · February 17, 2022, 6:31pm

Hi!
Logstash version: 7.17.0
Could you help me with logstash-mongodb issue, in view of input plugin for Logstash.
I work on some task, where I need to get logs from postgre and mongo.
Regarding to postgre - everything is fine.
But with mongo I faced with lot of issues.
First i tried to work with logstash-input-mongodb, probably it was great solution.
Unfortunately there is no support for many years, and it's difficult to use it with current versions of software.

https://github.com/phutchins/logstash-input-mongodb

Second way - jdbc input.
I tried different drivers.
mongodb-jdbc-1.0.3.jar
mongojdbc1.6.jar
mongojdbc4.0.jar

But still got NoClassDefFoundError.

Please, help me to fix it or probably there is another solution to send collection data from mongodb to Elasticsearch.

conf.d/mongo.conf

input {
        jdbc {
                jdbc_driver_library => "/etc/logstash/mongojdbc4.0.jar"
                jdbc_driver_class => "com.dbschema.MongoJdbcDriver"
                jdbc_connection_string => "jdbc:mongodb://username:password@127.0.0.1:27017/mydb"
                jdbc_user => "username"
                jdbc_password => "password"
                statement => "db.final_fields.find({},{'_id': false});"
  }
}

output {
        stdout {
                codec => rubydebug
        }
}

sudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/mongo.conf

Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2022-02-18 00:14:36.580 [main] runner - Starting Logstash {"logstash.version"=>"7.17.0", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
[INFO ] 2022-02-18 00:14:36.584 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true]
[WARN ] 2022-02-18 00:14:36.761 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2022-02-18 00:14:37.482 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[INFO ] 2022-02-18 00:14:38.335 [Converge PipelineAction::Create<main>] Reflections - Reflections took 50 ms to scan 1 urls, producing 119 keys and 417 values 
[WARN ] 2022-02-18 00:14:38.889 [Converge PipelineAction::Create<main>] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-02-18 00:14:38.911 [Converge PipelineAction::Create<main>] jdbc - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2022-02-18 00:14:39.080 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/mongo.conf"], :thread=>"#<Thread:0x486c78d2 run>"}
[INFO ] 2022-02-18 00:14:39.572 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>0.49}
[INFO ] 2022-02-18 00:14:39.635 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2022-02-18 00:14:39.674 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[ERROR] 2022-02-18 00:14:40.013 [[main]<jdbc] jdbc - Unable to connect to database. Tried 1 times {:message=>"Java::JavaLang::NoClassDefFoundError: com/mongodb/MongoClientURI", :exception=>Sequel::DatabaseConnectionError, :cause=>java.lang.NoClassDefFoundError: com/mongodb/MongoClientURI, :backtrace=>["com.dbschema.mongo.JMongoClient.<init>(com/dbschema/mongo/JMongoClient.java:27)", "com.dbschema.mongo.MongoService.<init>(com/dbschema/mongo/MongoService.java:28)", "com.dbschema.MongoJdbcDriver.connect(com/dbschema/MongoJdbcDriver.java:56)", "java.sql.DriverManager.getConnection(java/sql/DriverManager.java:677)", "java.sql.DriverManager.getConnection(java/sql/DriverManager.java:251)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:62)", "jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:566)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:426)", "org.jruby.javasupport.JavaMethod.invokeStaticDirect(org/jruby/javasupport/JavaMethod.java:358)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_52_dot_0.lib.sequel.adapters.jdbc.connect(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.52.0/lib/sequel/adapters/jdbc.rb:225)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_52_dot_0.lib.sequel.connection_pool.make_new(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.52.0/lib/sequel/connection_pool.rb:122)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_52_dot_0.lib.sequel.connection_pool.threaded.assign_connection(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.52.0/lib/sequel/connection_pool/threaded.rb:209)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_52_dot_0.lib.sequel.connection_pool.threaded.acquire(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.52.0/lib/sequel/connection_pool/threaded.rb:139)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_52_dot_0.lib.sequel.connection_pool.threaded.hold(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.52.0/lib/sequel/connection_pool/threaded.rb:91)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_52_dot_0.lib.sequel.database.connecting.synchronize(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.52.0/lib/sequel/database/connecting.rb:269)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_52_dot_0.lib.sequel.database.connecting.test_connection(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.52.0/lib/sequel/database/connecting.rb:278)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_52_dot_0.lib.sequel.database.misc.initialize(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.52.0/lib/sequel/database/misc.rb:175)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_52_dot_0.lib.sequel.database.connecting.connect(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.52.0/lib/sequel/database/connecting.rb:57)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_52_dot_0.lib.sequel.core.connect(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.52.0/lib/sequel/core.rb:124)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_2_dot_2.lib.logstash.plugin_mixins.jdbc.jdbc.jdbc_connect(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.2.2/lib/logstash/plugin_mixins/jdbc/jdbc.rb:117)", "org.jruby.RubyKernel.loop(org/jruby/RubyKernel.java:1442)", "org.jruby.RubyKernel$INVOKER$s$0$0$loop.call(org/jruby/RubyKernel$INVOKER$s$0$0$loop.gen)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_2_dot_2.lib.logstash.plugin_mixins.jdbc.jdbc.jdbc_connect(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.2.2/lib/logstash/plugin_mixins/jdbc/jdbc.rb:114)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_2_dot_2.lib.logstash.plugin_mixins.jdbc.jdbc.open_jdbc_connection(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.2.2/lib/logstash/plugin_mixins/jdbc/jdbc.rb:157)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_2_dot_2.lib.logstash.plugin_mixins.jdbc.jdbc.execute_statement(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.2.2/lib/logstash/plugin_mixins/jdbc/jdbc.rb:214)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_2_dot_2.lib.logstash.inputs.jdbc.execute_query(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.2.2/lib/logstash/inputs/jdbc.rb:339)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_2_dot_2.lib.logstash.inputs.jdbc.run(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.2.2/lib/logstash/inputs/jdbc.rb:307)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.inputworker(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:409)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_input(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:400)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:318)", "java.lang.Thread.run(java/lang/Thread.java:829)"]}
[WARN ] 2022-02-18 00:14:40.017 [[main]<jdbc] jdbc - Exception when executing JDBC query {:exception=>Sequel::DatabaseConnectionError, :message=>"Java::JavaLang::NoClassDefFoundError: com/mongodb/MongoClientURI", :cause=>"java.lang.NoClassDefFoundError: com/mongodb/MongoClientURI"}
[INFO ] 2022-02-18 00:14:40.159 [[main]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>"main"}
[INFO ] 2022-02-18 00:14:40.220 [Converge PipelineAction::Delete<main>] pipelinesregistry - Removed pipeline from registry successfully {:pipeline_id=>:main}
[INFO ] 2022-02-18 00:14:40.253 [LogStash::Runner] runner - Logstash shut down.

Tomo_M · February 20, 2022, 12:35pm

Though I'm not familiar with MongoDB, last resort could be using Exec input plugin with creating some bash script which retrieve data from MongoDB.

Bash script could be something like:

 #!/bin/bash
 mongo --quiet << 'EOT'
 use db01;
 db.coll_01.                             // collection 選択
    find({type:'A'},{_id:0}).            // typeが'A'のレコード
    sort({_id:-1}).                      // _idで逆向きにソート
    limit(3).                            // 個数制限
    toArray().reverse().                 // 反転したリスト取得
    forEach(S=>print(JSON.stringify(S)));// 要素表示
 EOT

The page just came up from a quick Google search. It is in Japanese and please use some translator if you want to read.

aaron-nimocks · February 20, 2022, 2:14pm

Try putting you .jar in the logstash-core library folder. Having it in other directories in the past cause this type of issues due to permissions I believe. Will be something like this depending on your version and OS. Then change your reference to that path and try again. If that doesn't work try completely removing the jdbc_driver_library setting from your input with the jar in the same folder as above.

/usr/share/logstash/logstash-core/lib/jars/

serfisar · February 28, 2022, 10:19am

Thank you, problem was with incorrect jdbc driver.
Currently i should configure the same function on logstash 2.4.1 version
And as I understand, there was another path not /usr/share/logstash/logstash-core/lib/jars/
Do you know path of driver libraries in old versions?

serfisar · February 28, 2022, 10:20am

Thank you, for alternative resolution

system · March 28, 2022, 10:20am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Jdbc input plugin Logstash	1	210	October 5, 2020
Logstash and mongodb connection using JDBC is not working Logstash	6	1173	March 16, 2020
JDBC input plugin error for MongoDB Logstash	2	1742	January 30, 2018
Logstash Mongo input error Logstash	1	318	January 16, 2023
Failed to configure JDBC Input Plugin with MongoDB Logstash	2	1566	September 10, 2017

Logstash and MongoDB input

Related topics