Mongodb/logstash connect error

Elastic Stack Logstash
1

This is an error message when running logstash.
error message:
Using bundled JDK: /home/admin/lg-862/jdk
Sending Logstash logs to /home/admin/lg-862/logs which is now configured via log4j2.properties
[2023-03-29T18:22:25,781][INFO ][logstash.runner ] Log4j configuration path used is: /home/admin/lg-862/config/log4j2.properties
[2023-03-29T18:22:25,789][WARN ][logstash.runner ] The use of JAVA_HOME has been deprecated. Logstash 8.0 and later ignores JAVA_HOME and uses the bundled JDK. Running Logstash with the bundled JDK is recommended. The bundled JDK has been verified to work with each specific version of Logstash, and generally provides best performance and reliability. If you have compelling reasons for using your own JDK (organizational-specific compliance requirements, for example), you can configure LS_JAVA_HOME to use that version instead.
[2023-03-29T18:22:25,790][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.6.2", "jruby.version"=>"jruby 9.3.10.0 (2.6.8) 2023-02-01 107b2e6697 OpenJDK 64-Bit Server VM 17.0.6+10 on 17.0.6+10 +indy +jit [x86_64-linux]"}
[2023-03-29T18:22:25,792][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2023-03-29T18:22:25,921][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2023-03-29T18:22:26,367][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2023-03-29T18:22:26,522][INFO ][org.reflections.Reflections] Reflections took 90 ms to scan 1 urls, producing 127 keys and 444 values
[2023-03-29T18:22:26,976][INFO ][logstash.javapipeline ] Pipeline main is configured with pipeline.ecs_compatibility: v8 setting. All plugins in this pipeline will default to ecs_compatibility => v8 unless explicitly configured otherwise.
[2023-03-29T18:22:26,996][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/home/admin/lg-862/workshop.conf"], :thread=>"#<Thread:0x2911692b@/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:131 run>"}
[2023-03-29T18:22:27,479][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>0.48}
[2023-03-29T18:22:27,587][ERROR][logstash.javapipeline ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<LogStash::ConfigurationError: Must set either :statement or :statement_filepath. Only one may be set at a time.>, :backtrace=>["/home/admin/lg-862/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/inputs/jdbc.rb:272:in `register'", "/home/admin/lg-862/vendor/bundle/jruby/2.6.0/gems/logstash-mixin-ecs_compatibility_support-1.3.0-java/lib/logstash/plugin_mixins/ecs_compatibility_support/target_check.rb:48:in `register'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:234:in `block in register_plugins'", "org/jruby/RubyArray.java:1865:in `each'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:233:in `register_plugins'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:392:in `start_inputs'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:317:in `start_workers'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:191:in `run'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:143:in `block in start'"], "pipeline.sources"=>["/home/admin/lg-862/workshop.conf"], :thread=>"#<Thread:0x2911692b@/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:131 run>"}
[2023-03-29T18:22:27,588][INFO ][logstash.javapipeline ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2023-03-29T18:22:27,590][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil}
[2023-03-29T18:22:27,599][INFO ][logstash.runner ] Logstash shut down.
[2023-03-29T18:22:27,603][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:790) ~[jruby.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:753) ~[jruby.jar:?]
at home.admin.lg_minus_862.lib.bootstrap.environment.(/home/admin/lg-862/lib/bootstrap/environment.rb:91) ~[?:?]

This is the conf file I ran.

input{
jdbc {
jdbc_connection_string => "jdbc:mongodb://XX.XX.XX.XX:27017/mytestdb"
jdbc_user => "admin"
jdbc_driver_library => "/home/admin/lg-862/logstash-core/lib/jars/mongo-java-driver-3.9.1-javadoc.jar"
jdbc_driver_class => "com.mongodb.MongoClient"
}
}

output{
stdout{ }
}

Is it possible to do logstash connection with mongoDB?

2

Have you tried specifying either sample SQL using statement or statement_path, as per the usage documentation?

3

Thank you for answer. I wrote a sample sql and the error message changed.

[ERROR][logstash.javapipeline ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<LogStash::PluginLoadingError: #<Java::JavaLang::ClassNotFoundException: com.mongodb.MongoClient>. Are you sure you've included the correct jdbc driver in :jdbc_driver_library?>, :backtrace=>["/home/admin/lg-862/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/plugin_mixins/jdbc/common.rb:45:in load_driver'", "/home/admin/lg-862/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/inputs/jdbc.rb:307:in register'", "/home/admin/lg-862/vendor/bundle/jruby/2.6.0/gems/logstash-mixin-ecs_compatibility_support-1.3.0-java/lib/logstash/plugin_mixins/ecs_compatibility_support/target_check.rb:48:in register'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:234:in block in register_plugins'", "org/jruby/RubyArray.java:1865:in each'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:233:in register_plugins'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:392:in start_inputs'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:317:in start_workers'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:191:in run'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:143:in block in start'"], "pipeline.sources"=>["/home/admin/lg-862/workshop.conf"], :thread=>"#<Thread:0x56ab17dc@/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:131 run>"}

It seems to be a library issue this time...
Do you have logstash-input-mongodb before that?

I saw the article you talked about on Slack before and I left a comment there, but it is not easy to connect logstash and mongodb.

4

So that error is talking about the JDBC driver, so I would recommend double checking you have the correct driver, version and path in the config.

If it's this thread you're referring to, there are several options listed for interacting with Elasticsearch aside from Logstash. Can you explain what you're trying to do and perhaps we can propose an alternative?

5

I want to connect MongoDB and Logstash to send data to Elasticsearch.

I want to use the ELK method, so I want Logstash and MongoDB to be connected.

It looks like you are not getting the right drivers.
Is it possible to get the mongodb jdbc driver?

conf file, but it doesn't seem to work.

I also tried using monstache among other methods, but the installation did not go smoothly.

6

Ok, have you double checked the driver to make sure you are using a version compatible with the version of MongoDB that you are trying to extract from?

I do have another option if you are on a later version of Elastic. If you're 8.5.0 or later there is a new MongoDB connector, albeit in technical preview, that can also be used to extract form MongoDB into Elasticsearch.

7

Thank you for answer.

All are using 8.6.2. I'm going to store log data in mongodb, can kibana handle its capacity?

I downloaded the driver again to try it with logstash. I am getting this error. can you solve it?

conf file:

input{
jdbc{
jdbc_driver_library => "/home/admin/lg-862/lgstash-core/lib/jars/mongojdbc4.8.jar"
jdbc_driver_class => "Java::com.wisecoders.dbschema.mongodb.JdbcDriver"
jdbc_connection_string => "mongodb://XXXXXX:XXXXXX@XXX.XXX.XX.XX:27017/mytestdb"
jdbc_user => "XXXXXXX"
jdbc_password => "XXXXX"
statement => "db.getCollection('testcoll').find({})"

}

}

output{
stdout{ }
}

error:

./bin/logstash -f workshop.conf
Using bundled JDK: /home/admin/lg-862/jdk
jruby: warning: unknown property jruby.version
Sending Logstash logs to /home/admin/lg-862/logs which is now configured via log4j2.properties
[2023-04-03T14:41:22,746][INFO ][logstash.runner ] Log4j configuration path used is: /home/admin/lg-862/config/log4j2.properties
[2023-04-03T14:41:22,752][WARN ][logstash.runner ] The use of JAVA_HOME has been deprecated. Logstash 8.0 and later ignores JAVA_HOME and uses the bundled JDK. Running Logstash with the bundled JDK is recommended. The bundled JDK has been verified to work with each specific version of Logstash, and generally provides best performance and reliability. If you have compelling reasons for using your own JDK (organizational-specific compliance requirements, for example), you can configure LS_JAVA_HOME to use that version instead.
[2023-04-03T14:41:22,752][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.6.2", "jruby.version"=>"jruby 9.3.10.0 (2.6.8) 2023-02-01 107b2e6697 OpenJDK 64-Bit Server VM 17.0.6+10 on 17.0.6+10 +jit [x86_64-linux]"}
[2023-04-03T14:41:22,754][INFO ][logstash.runner ] JVM bootstrap flags: [-Djruby.version=9.2.13.0, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2023-04-03T14:41:22,887][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2023-04-03T14:41:23,254][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2023-04-03T14:41:23,485][INFO ][org.reflections.Reflections] Reflections took 102 ms to scan 1 urls, producing 127 keys and 444 values
[2023-04-03T14:41:23,888][INFO ][logstash.javapipeline ] Pipeline main is configured with pipeline.ecs_compatibility: v8 setting. All plugins in this pipeline will default to ecs_compatibility => v8 unless explicitly configured otherwise.
[2023-04-03T14:41:23,907][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/home/admin/lg-862/workshop.conf"], :thread=>"#<Thread:0x6d34d575@/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:131 run>"}
[2023-04-03T14:41:24,409][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>0.5}
[2023-04-03T14:41:24,633][ERROR][logstash.javapipeline ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<LogStash::PluginLoadingError: unable to load /home/admin/lg-862/lgstash-core/lib/jars/mongojdbc4.8.jar from :jdbc_driver_library, file not readable (please check user and group permissions for the path)>, :backtrace=>["/home/admin/lg-862/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/plugin_mixins/jdbc/common.rb:59:in `block in load_driver_jars'", "org/jruby/RubyArray.java:1865:in `each'", "/home/admin/lg-862/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/plugin_mixins/jdbc/common.rb:54:in `load_driver_jars'", "/home/admin/lg-862/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/plugin_mixins/jdbc/common.rb:34:in `load_driver'", "/home/admin/lg-862/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/inputs/jdbc.rb:307:in `register'", "/home/admin/lg-862/vendor/bundle/jruby/2.6.0/gems/logstash-mixin-ecs_compatibility_support-1.3.0-java/lib/logstash/plugin_mixins/ecs_compatibility_support/target_check.rb:48:in `register'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:234:in `block in register_plugins'", "org/jruby/RubyArray.java:1865:in `each'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:233:in `register_plugins'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:392:in `start_inputs'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:317:in `start_workers'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:191:in `run'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:143:in `block in start'"], "pipeline.sources"=>["/home/admin/lg-862/workshop.conf"], :thread=>"#<Thread:0x6d34d575@/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:131 run>"}
[2023-04-03T14:41:24,634][INFO ][logstash.javapipeline ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2023-04-03T14:41:24,641][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil}
[2023-04-03T14:41:24,650][INFO ][logstash.runner ] Logstash shut down.
[2023-04-03T14:41:24,654][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:790) ~[jruby.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:753) ~[jruby.jar:?]
at home.admin.lg_minus_862.lib.bootstrap.environment.(/home/admin/lg-862/lib/bootstrap/environment.rb:91) ~[?:?]

8

I am not using elastic cloud.

9

Looks like the process doesn't have sufficient entitlements to access the driver. Can you check the permissions?

10

conf file:

input{
jdbc{
jdbc_driver_class => "Java::com.wisecoders.dbschema.mongodb.JdbcDriver"
jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/mongojdbc4.8.jar"
jdbc_connection_string => "mongodb://XXX:XXX@XX.XX.XX.XX:27017/mytestdb"
jdbc_user => "XX"
jdbc_password => "XX"
statement => "db.getCollection('testcoll').find({})"

}

}

output{
stdout{ }
}

error :

[2023-04-03T17:52:15,435][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>0.48}
[2023-04-03T17:52:15,574][ERROR][logstash.inputs.jdbc ][main] Unable to connect to database. Tried 1 times {:message=>"LoadError: no such file to load -- sequel/adapters/mongodb", :exception=>Sequel::AdapterNotFound, :cause=>#<LoadError: no such file to load -- sequel/adapters/mongodb>, :backtrace=>["org/jruby/RubyKernel.java:1017:in require'", "/home/admin/lg-862/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/connecting.rb:87:in load_adapter'", "/home/admin/lg-862/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/connecting.rb:17:in adapter_class'", "/home/admin/lg-862/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/connecting.rb:36:in connect'", "/home/admin/lg-862/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/core.rb:124:in connect'", "/home/admin/lg-862/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/plugin_mixins/jdbc/jdbc.rb:118:in block in jdbc_connect'", "org/jruby/RubyKernel.java:1507:in loop'", "/home/admin/lg-862/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/plugin_mixins/jdbc/jdbc.rb:115:in jdbc_connect'", "/home/admin/lg-862/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/plugin_mixins/jdbc/jdbc.rb:158:in open_jdbc_connection'", "/home/admin/lg-862/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/inputs/jdbc.rb:309:in register'", "/home/admin/lg-862/vendor/bundle/jruby/2.6.0/gems/logstash-mixin-ecs_compatibility_support-1.3.0-java/lib/logstash/plugin_mixins/ecs_compatibility_support/target_check.rb:48:in register'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:234:in block in register_plugins'", "org/jruby/RubyArray.java:1865:in each'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:233:in register_plugins'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:392:in start_inputs'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:317:in start_workers'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:191:in run'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:143:in block in start'"]}
[2023-04-03T17:52:15,649][ERROR][logstash.javapipeline ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<Sequel::AdapterNotFound: LoadError: no such file to load -- sequel/adapters/mongodb>, :backtrace=>["org/jruby/RubyKernel.java:1017:in require'", "/home/admin/lg-862/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/connecting.rb:87:in load_adapter'", "/home/admin/lg-862/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/connecting.rb:17:in adapter_class'", "/home/admin/lg-862/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/connecting.rb:36:in connect'", "/home/admin/lg-862/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/core.rb:124:in connect'", "/home/admin/lg-862/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/plugin_mixins/jdbc/jdbc.rb:118:in block in jdbc_connect'", "org/jruby/RubyKernel.java:1507:in loop'", "/home/admin/lg-862/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/plugin_mixins/jdbc/jdbc.rb:115:in jdbc_connect'", "/home/admin/lg-862/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/plugin_mixins/jdbc/jdbc.rb:158:in open_jdbc_connection'", "/home/admin/lg-862/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/inputs/jdbc.rb:309:in register'", "/home/admin/lg-862/vendor/bundle/jruby/2.6.0/gems/logstash-mixin-ecs_compatibility_support-1.3.0-java/lib/logstash/plugin_mixins/ecs_compatibility_support/target_check.rb:48:in register'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:234:in block in register_plugins'", "org/jruby/RubyArray.java:1865:in each'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:233:in register_plugins'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:392:in start_inputs'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:317:in start_workers'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:191:in run'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:143:in block in start'"], "pipeline.sources"=>["/home/admin/lg-862/workshop.conf"], :thread=>"#<Thread:0x6c472cb2@/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:131 run>"}
[2023-04-03T17:52:15,649][INFO ][logstash.javapipeline ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2023-04-03T17:52:15,652][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil}
[2023-04-03T17:52:15,661][INFO ][logstash.runner ] Logstash shut down.
[2023-04-03T17:52:15,664][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:790) ~[jruby.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:753) ~[jruby.jar:?]
at home.admin.lg_minus_862.lib.bootstrap.environment.(/home/admin/lg-862/lib/bootstrap/environment.rb:91) ~[?:?]
admin@qa-optool01-svr:~/lg-862$

When I changed it after reading the article to remove jdbc_driver_library or specify it as the path written above, another error message appeared...

help..

11

LoadError: no such file to load -- sequel/adapters/mongodb

This problem is too hard for me..

12

It seems to be a version issue. ELK is all using version 8.6.2, but mongoDB Driver is version 4.8.

13

Can you share where you got the driver from? I've found this older thread talking about driver compatibility, so I wonder if you are using the correct driver.

14

1.curl -O https://dbschema.com/jdbc-drivers/MongoDbJdbcDriver.zip
2.unzip MongoDbJdbcDriver.zip

I have downloaded and used this.

confile:

input{
jdbc{
jdbc_driver_class => "com.mongodb.jdbc.MongoDriver"
jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/mongodb-driver-core-4.8.1.jar"
jdbc_connection_string => "jdbc:mongodb://XXX:XXX@XXX.XXX.xXX.XXX:27017/mytestdb"
jdbc_user => "XXXXXXX"
jdbc_password => "XXXXXX"
statement => "db.getCollection('testcoll').find({})"

}

}

output{
stdout{ }
}

Currently I am writing like this

[ERROR][logstash.javapipeline ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<LogStash::PluginLoadingError: #<Java::JavaLang::ClassNotFoundException: com.mongodb.jdbc.MongoDriver>. Are you sure you've included the correct jdbc driver in :jdbc_driver_library?>, :backtrace=>["/home/admin/lg-862/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/plugin_mixins/jdbc/common.rb:45:in load_driver'", "/home/admin/lg-862/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/inputs/jdbc.rb:307:in register'", "/home/admin/lg-862/vendor/bundle/jruby/2.6.0/gems/logstash-mixin-ecs_compatibility_support-1.3.0-java/lib/logstash/plugin_mixins/ecs_compatibility_support/target_check.rb:48:in register'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:234:in block in register_plugins'", "org/jruby/RubyArray.java:1865:in each'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:233:in register_plugins'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:392:in start_inputs'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:317:in start_workers'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:191:in run'", "/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:143:in block in start'"], "pipeline.sources"=>["/home/admin/lg-862/workshop.conf"], :thread=>"#<Thread:0x100c82d@/home/admin/lg-862/logstash-core/lib/logstash/java_pipeline.rb:131 run>"}
[2023-04-03T21:25:28,124][INFO ][logstash.javapipeline ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2023-04-03T21:25:28,132][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil}

I am in this error state.

15

I've downloaded a copy of the driver zip you are currently using. Should the jdbc_driver_library property not point to jar mongojdbc4.8.jar rather than mongodb-driver-core-4.8.1.jar? Looking at this related thread I suspect that might be the problem.

16

Thank you for answer.

I paid attention to that part and corrected it, and then I encountered another error.

I wonder if I'm still writing the conf file incorrectly...
I'm having a hard time figuring out what the problem is.

com.mongodb.MongoSocketOpenException: Exception opening socket
        at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:73) ~[mongodb-driver-core-4.8.1.jar:?]
        at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:183) ~[mongodb-driver-core-4.8.1.jar:?]
        at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.lookupServerDescription(DefaultServerMonitor.java:198) [mongodb-driver-core-4.8.1.jar:?]
        at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:158) [mongodb-driver-core-4.8.1.jar:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
        at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
        at sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:542) ~[?:?]
        at sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) ~[?:?]
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) ~[?:?]
        at java.net.Socket.connect(Socket.java:633) ~[?:?]
        at com.mongodb.internal.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:107) ~[mongodb-driver-core-4.8.1.jar:?]
        at com.mongodb.internal.connection.SocketStream.initializeSocket(SocketStream.java:82) ~[mongodb-driver-core-4.8.1.jar:?]
        at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:68) ~[mongodb-driver-core-4.8.1.jar:?]
        ... 4 more
17

This looks like Logstash is struggling to connect to your MongoDB instance. Can you check your connection URL, username and password?

18

you're right.
I did have a connection.

Do you know what to do if you want to move only new data to Elasticsearch when data piles up in mongodb?

Now, the existing data is sent and it is terminated.
Or Pipeline terminated {"pipeline.id"=>"main"}
I leave this message and stay still, but there is no reaction when new data is added.

19

If you don't specify a value for schedule the pipeline will be executed once. Take a look at the below resources in the documentation:

  1. JDBC Input Plugin- Scheduling
  2. schedule parameter
20

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.