# Logstash and MongoDB input

**URL:** <https://discuss.elastic.co/t/logstash-and-mongodb-input/297535>\
**Category:** Logstash\
**Created:** [February 17, 2022, 6:31pm UTC](https://discuss.elastic.co/t/logstash-and-mongodb-input/297535 "2022-02-17T18:31:48Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![serfisar](https://avatars.discourse-cdn.com/v4/letter/s/ac91a4/32.png) [@serfisar](https://discuss.elastic.co/u/serfisar)\
**Post date:** [February 17, 2022, 6:31pm UTC](https://discuss.elastic.co/t/logstash-and-mongodb-input/297535/1 "2022-02-17T18:31:48Z")

</div>

Hi!  
Logstash version: 7.17.0  
Could you help me with logstash-mongodb issue, in view of input plugin for Logstash.  
I work on some task, where I need to get logs from postgre and mongo.  
Regarding to postgre - everything is fine.  
But with mongo I faced with lot of issues.  
First i tried to work with logstash-input-mongodb, probably it was great solution.  
Unfortunately there is no support for many years, and it's difficult to use it with current versions of software.

[https://github.com/phutchins/logstash-input-mongodb](https://github.com/phutchins/logstash-input-mongodb)

Second way - jdbc input.  
I tried different drivers.  
mongodb-jdbc-1.0.3.jar  
mongojdbc1.6.jar  
mongojdbc4.0.jar

But still got NoClassDefFoundError.

Please, help me to fix it or probably there is another solution to send collection data from mongodb to Elasticsearch.

conf.d/mongo.conf

```auto
input {
        jdbc {
                jdbc_driver_library => "/etc/logstash/mongojdbc4.0.jar"
                jdbc_driver_class => "com.dbschema.MongoJdbcDriver"
                jdbc_connection_string => "jdbc:mongodb://username:password@127.0.0.1:27017/mydb"
                jdbc_user => "username"
                jdbc_password => "password"
                statement => "db.final_fields.find({},{'_id': false});"
  }
}

output {
        stdout {
                codec => rubydebug
        }
}

```

sudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/mongo.conf

```auto
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO] 2022-02-18 00:14:36.580 [main] runner - Starting Logstash {"logstash.version"=>"7.17.0", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
[INFO] 2022-02-18 00:14:36.584 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true]
[WARN] 2022-02-18 00:14:36.761 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO] 2022-02-18 00:14:37.482 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[INFO] 2022-02-18 00:14:38.335 [Converge PipelineAction::Create<main>] Reflections - Reflections took 50 ms to scan 1 urls, producing 119 keys and 417 values 
[WARN] 2022-02-18 00:14:38.889 [Converge PipelineAction::Create<main>] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN] 2022-02-18 00:14:38.911 [Converge PipelineAction::Create<main>] jdbc - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO] 2022-02-18 00:14:39.080 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/mongo.conf"], :thread=>"#<Thread:0x486c78d2 run>"}
[INFO] 2022-02-18 00:14:39.572 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>0.49}
[INFO] 2022-02-18 00:14:39.635 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO] 2022-02-18 00:14:39.674 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[ERROR] 2022-02-18 00:14:40.013 [[main]<jdbc] jdbc - Unable to connect to database. Tried 1 times {:message=>"Java::JavaLang::NoClassDefFoundError: com/mongodb/MongoClientURI", :exception=>Sequel::DatabaseConnectionError, :cause=>java.lang.NoClassDefFoundError: com/mongodb/MongoClientURI, :backtrace=>["com.dbschema.mongo.JMongoClient.<init>(com/dbschema/mongo/JMongoClient.java:27)", "com.dbschema.mongo.MongoService.<init>(com/dbschema/mongo/MongoService.java:28)", "com.dbschema.MongoJdbcDriver.connect(com/dbschema/MongoJdbcDriver.java:56)", "java.sql.DriverManager.getConnection(java/sql/DriverManager.java:677)", "java.sql.DriverManager.getConnection(java/sql/DriverManager.java:251)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:62)", "jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:566)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:426)", "org.jruby.javasupport.JavaMethod.invokeStaticDirect(org/jruby/javasupport/JavaMethod.java:358)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_52_dot_0.lib.sequel.adapters.jdbc.connect(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.52.0/lib/sequel/adapters/jdbc.rb:225)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_52_dot_0.lib.sequel.connection_pool.make_new(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.52.0/lib/sequel/connection_pool.rb:122)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_52_dot_0.lib.sequel.connection_pool.threaded.assign_connection(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.52.0/lib/sequel/connection_pool/threaded.rb:209)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_52_dot_0.lib.sequel.connection_pool.threaded.acquire(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.52.0/lib/sequel/connection_pool/threaded.rb:139)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_52_dot_0.lib.sequel.connection_pool.threaded.hold(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.52.0/lib/sequel/connection_pool/threaded.rb:91)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_52_dot_0.lib.sequel.database.connecting.synchronize(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.52.0/lib/sequel/database/connecting.rb:269)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_52_dot_0.lib.sequel.database.connecting.test_connection(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.52.0/lib/sequel/database/connecting.rb:278)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_52_dot_0.lib.sequel.database.misc.initialize(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.52.0/lib/sequel/database/misc.rb:175)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_52_dot_0.lib.sequel.database.connecting.connect(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.52.0/lib/sequel/database/connecting.rb:57)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_52_dot_0.lib.sequel.core.connect(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.52.0/lib/sequel/core.rb:124)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_2_dot_2.lib.logstash.plugin_mixins.jdbc.jdbc.jdbc_connect(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.2.2/lib/logstash/plugin_mixins/jdbc/jdbc.rb:117)", "org.jruby.RubyKernel.loop(org/jruby/RubyKernel.java:1442)", "org.jruby.RubyKernel$INVOKER$s$0$0$loop.call(org/jruby/RubyKernel$INVOKER$s$0$0$loop.gen)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_2_dot_2.lib.logstash.plugin_mixins.jdbc.jdbc.jdbc_connect(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.2.2/lib/logstash/plugin_mixins/jdbc/jdbc.rb:114)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_2_dot_2.lib.logstash.plugin_mixins.jdbc.jdbc.open_jdbc_connection(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.2.2/lib/logstash/plugin_mixins/jdbc/jdbc.rb:157)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_2_dot_2.lib.logstash.plugin_mixins.jdbc.jdbc.execute_statement(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.2.2/lib/logstash/plugin_mixins/jdbc/jdbc.rb:214)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_2_dot_2.lib.logstash.inputs.jdbc.execute_query(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.2.2/lib/logstash/inputs/jdbc.rb:339)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_2_dot_2.lib.logstash.inputs.jdbc.run(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.2.2/lib/logstash/inputs/jdbc.rb:307)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.inputworker(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:409)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_input(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:400)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:318)", "java.lang.Thread.run(java/lang/Thread.java:829)"]}
[WARN] 2022-02-18 00:14:40.017 [[main]<jdbc] jdbc - Exception when executing JDBC query {:exception=>Sequel::DatabaseConnectionError, :message=>"Java::JavaLang::NoClassDefFoundError: com/mongodb/MongoClientURI", :cause=>"java.lang.NoClassDefFoundError: com/mongodb/MongoClientURI"}
[INFO] 2022-02-18 00:14:40.159 [[main]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>"main"}
[INFO] 2022-02-18 00:14:40.220 [Converge PipelineAction::Delete<main>] pipelinesregistry - Removed pipeline from registry successfully {:pipeline_id=>:main}
[INFO] 2022-02-18 00:14:40.253 [LogStash::Runner] runner - Logstash shut down.

```

---

<div class="post-metadata">

**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\
**Post date:** [February 20, 2022, 12:35pm UTC](https://discuss.elastic.co/t/logstash-and-mongodb-input/297535/2 "2022-02-20T12:35:05Z")

</div>

Though I'm not familiar with MongoDB, last resort could be using [Exec input plugin](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-exec.html) with creating some bash script which retrieve data from MongoDB.

Bash script could be something like:

```auto
 #!/bin/bash
 mongo --quiet << 'EOT'
 use db01;
 db.coll_01. // collection 選択
    find({type:'A'},{_id:0}). // typeが'A'のレコード
    sort({_id:-1}). // _idで逆向きにソート
    limit(3). // 個数制限
    toArray().reverse(). // 反転したリスト取得
    forEach(S=>print(JSON.stringify(S)));// 要素表示
 EOT

```

> **[◇mongo-shell script実行(here document) - 万象酔歩](http://k-hiura.cocolog-nifty.com/blog/2020/08/post-b77d4b.html)**
>
> \#!/bin/bash mongo --quiet \<&...

The page just came up from a quick Google search. It is in Japanese and please use some translator if you want to read.

---

<div class="post-metadata">

**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\
**Post date:** [February 20, 2022, 2:14pm UTC](https://discuss.elastic.co/t/logstash-and-mongodb-input/297535/3 "2022-02-20T14:14:39Z")

</div>

Try putting you .jar in the logstash-core library folder. Having it in other directories in the past cause this type of issues due to permissions I believe. Will be something like this depending on your version and OS. Then change your reference to that path and try again. If that doesn't work try completely removing the `jdbc_driver_library` setting from your input with the jar in the same folder as above.

`/usr/share/logstash/logstash-core/lib/jars/`

---

<div class="post-metadata">

**Author:** ![serfisar](https://avatars.discourse-cdn.com/v4/letter/s/ac91a4/32.png) [@serfisar](https://discuss.elastic.co/u/serfisar)\
**Post date:** [February 28, 2022, 10:19am UTC](https://discuss.elastic.co/t/logstash-and-mongodb-input/297535/4 "2022-02-28T10:19:33Z")

</div>

Thank you, problem was with incorrect jdbc driver.  
Currently i should configure the same function on logstash 2.4.1 version  
And as I understand, there was another path not /usr/share/logstash/logstash-core/lib/jars/  
Do you know path of driver libraries in old versions?

---

<div class="post-metadata">

**Author:** ![serfisar](https://avatars.discourse-cdn.com/v4/letter/s/ac91a4/32.png) [@serfisar](https://discuss.elastic.co/u/serfisar)\
**Post date:** [February 28, 2022, 10:20am UTC](https://discuss.elastic.co/t/logstash-and-mongodb-input/297535/5 "2022-02-28T10:20:15Z")

</div>

Thank you, for alternative resolution

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 28, 2022, 10:20am UTC](https://discuss.elastic.co/t/logstash-and-mongodb-input/297535/6 "2022-03-28T10:20:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.