Hi
I need to import my documents from mongodb to Elasticsearch and I used logstash.
This is my output in CMD when I run this command:
C:\logstash-7.16.3\bin\logstash -f c:\logstash-7.16.3\bin\mongo2.conf
"Using bundled JDK: ."
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to C:/logstash-7.16.3/logs which is now configured via log4j2.properties
[2022-01-19T09:41:29,496][INFO ][logstash.runner ] Log4j configuration path used is: C:\logstash-7.16.3\config\log4j2.properties
[2022-01-19T09:41:29,504][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.16.3", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [mswin32-x86_64]"}
[2022-01-19T09:41:29,592][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-01-19T09:41:30,922][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-01-19T09:41:31,744][INFO ][org.reflections.Reflections] Reflections took 84 ms to scan 1 urls, producing 119 keys and 417 values
[2022-01-19T09:41:32,816][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2022-01-19T09:41:33,061][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2022-01-19T09:41:33,226][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2022-01-19T09:41:33,238][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (7.16.3) {:es_version=>7}
[2022-01-19T09:41:33,240][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2022-01-19T09:41:33,339][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[2022-01-19T09:41:33,369][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["c:/logstash-7.16.3/bin/mongo2.conf"], :thread=>"#<Thread:0x7404b206 run>"}
[2022-01-19T09:41:34,023][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>0.65}
[2022-01-19T09:41:34,107][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-01-19T09:41:34,186][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
java.nio.file.NoSuchFileException: C:\Users\Saeed\.DbSchema\logs\MongoDbJdbcDriver.log.lck
at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:85)
at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
at java.base/sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:120)
at java.base/java.nio.channels.FileChannel.open(FileChannel.java:292)
at java.base/java.nio.channels.FileChannel.open(FileChannel.java:345)
at java.logging/java.util.logging.FileHandler.openFiles(FileHandler.java:512)
at java.logging/java.util.logging.FileHandler.<init>(FileHandler.java:308)
at com.dbschema.MongoJdbcDriver.<clinit>(MongoJdbcDriver.java:35)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:398)
at org.jruby.javasupport.JavaSupportImpl.loadJavaClass(JavaSupportImpl.java:157)
at org.jruby.javasupport.Java.getProxyClassOrNull(Java.java:961)
at org.jruby.javasupport.Java.getProxyClassOrNull(Java.java:948)
at org.jruby.javasupport.Java.getProxyOrPackageUnderPackage(Java.java:905)
at org.jruby.javasupport.JavaPackage.method_missing(JavaPackage.java:252)
at org.jruby.javasupport.JavaPackage$INVOKER$i$method_missing.call(JavaPackage$INVOKER$i$method_missing.gen)
at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:833)
at org.jruby.runtime.Helpers$MethodMissingMethod.call(Helpers.java:591)
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:196)
at org.jruby.runtime.callsite.CachingCallSite.callMethodMissing(CachingCallSite.java:440)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:352)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:144)
at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:345)
at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)
at org.jruby.ir.interpreter.Interpreter.INTERPRET_EVAL(Interpreter.java:106)
at org.jruby.ir.interpreter.Interpreter.evalCommon(Interpreter.java:158)
at org.jruby.ir.interpreter.Interpreter.evalWithBinding(Interpreter.java:182)
at org.jruby.RubyKernel.evalCommon(RubyKernel.java:1086)
at org.jruby.RubyKernel.eval(RubyKernel.java:1048)
at org.jruby.RubyKernel$INVOKER$s$0$3$eval.call(RubyKernel$INVOKER$s$0$3$eval.gen)
at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)
at C_3a_.logstash_minus_7_dot_16_dot_3.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_51_dot_0.lib.sequel.adapters.jdbc.RUBY$method$load_driver$0(C:/logstash-7.16.3/vendor/bundle/jruby/2.5.0/gems/sequel-5.51.0/lib/sequel/adapters/jdbc.rb:55)
at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)
at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)
at C_3a_.logstash_minus_7_dot_16_dot_3.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_1_dot_8.lib.logstash.plugin_mixins.jdbc.common.RUBY$method$load_driver$0(C:/logstash-7.16.3/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.1.8/lib/logstash/plugin_mixins/jdbc/common.rb:33)
at C_3a_.logstash_minus_7_dot_16_dot_3.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_1_dot_8.lib.logstash.plugin_mixins.jdbc.common.RUBY$method$load_driver$0$__VARARGS__(C:/logstash-7.16.3/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.1.8/lib/logstash/plugin_mixins/jdbc/common.rb:20)
at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)
at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)
at C_3a_.logstash_minus_7_dot_16_dot_3.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_1_dot_8.lib.logstash.inputs.jdbc.RUBY$method$run$0(C:/logstash-7.16.3/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.1.8/lib/logstash/inputs/jdbc.rb:294)
at C_3a_.logstash_minus_7_dot_16_dot_3.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_1_dot_8.lib.logstash.inputs.jdbc.RUBY$method$run$0$__VARARGS__(C:/logstash-7.16.3/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.1.8/lib/logstash/inputs/jdbc.rb:293)
at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)
at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)
at C_3a_.logstash_minus_7_dot_16_dot_3.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$inputworker$0(C:/logstash-7.16.3/logstash-core/lib/logstash/java_pipeline.rb:409)
at C_3a_.logstash_minus_7_dot_16_dot_3.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$inputworker$0$__VARARGS__(C:/logstash-7.16.3/logstash-core/lib/logstash/java_pipeline.rb:404)
at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)
at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)
at C_3a_.logstash_minus_7_dot_16_dot_3.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_input$1(C:/logstash-7.16.3/logstash-core/lib/logstash/java_pipeline.rb:400)
at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)
at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)
at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)
at org.jruby.runtime.Block.call(Block.java:139)
at org.jruby.RubyProc.call(RubyProc.java:318)
at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)
at java.base/java.lang.Thread.run(Thread.java:829)
[2022-01-19T09:41:34,711][INFO ][org.mongodb.driver.cluster][main][812be7dc7d2bb2fa29f678e2d7a580e3cfdcbc14936292f801a2db8d55e183ec] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=500}
[2022-01-19T09:41:34,754][INFO ][org.mongodb.driver.cluster][main][812be7dc7d2bb2fa29f678e2d7a580e3cfdcbc14936292f801a2db8d55e183ec] Cluster description not yet available. Waiting for 30000 ms before timing out
[2022-01-19T09:41:34,771][INFO ][org.mongodb.driver.connection][main][812be7dc7d2bb2fa29f678e2d7a580e3cfdcbc14936292f801a2db8d55e183ec] Opened connection [connectionId{localValue:1, serverValue:116}] to localhost:27017
[2022-01-19T09:41:34,777][INFO ][org.mongodb.driver.cluster][main][812be7dc7d2bb2fa29f678e2d7a580e3cfdcbc14936292f801a2db8d55e183ec] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 4, 6]}, minWireVersion=0, maxWireVersion=9, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=3563900}
[2022-01-19T09:41:34,802][INFO ][org.mongodb.driver.connection][main][812be7dc7d2bb2fa29f678e2d7a580e3cfdcbc14936292f801a2db8d55e183ec] Opened connection [connectionId{localValue:2, serverValue:117}] to localhost:27017
Jan 19, 2022 9:41:34 AM com.dbschema.MongoPreparedStatement executeQuery
INFO: Execute db.getCollection('city').find({})
[To redirect Truffle log output to a file use one of the following options:
* '--log.file=<path>' if the option is passed using a guest language launcher.
* '-Dpolyglot.log.file=<path>' if the option is passed using the host Java launcher.
* Configure logging using the polyglot embedding API.]
[engine] WARNING: The polyglot context is using an implementation that does not support runtime compilation.
The guest application code will therefore be executed in interpreted mode only.
Execution only in interpreted mode will strongly impact the guest application performance.
For more information on using GraalVM see https://www.graalvm.org/java/quickstart/.
To disable this warning the '--engine.WarnInterpreterOnly=false' option or use the '-Dpolyglot.engine.WarnInterpreterOnly=false' system property.
Jan 19, 2022 9:41:35 AM com.dbschema.wrappers.WrappedMongoDatabase$GetCollectionProxyExecutable execute
INFO: Get collection city com.mongodb.client.internal.MongoCollectionImpl@22bd42d5
[2022-01-19T09:41:35,811][INFO ][logstash.inputs.jdbc ][main][812be7dc7d2bb2fa29f678e2d7a580e3cfdcbc14936292f801a2db8d55e183ec] (0.867947s) db.getCollection('city').find({})
[2022-01-19T09:41:35,874][WARN ][logstash.inputs.jdbc ][main][812be7dc7d2bb2fa29f678e2d7a580e3cfdcbc14936292f801a2db8d55e183ec] Exception when executing JDBC query {:exception=>"Java::OrgLogstash::MissingConverterException: Missing Converter handling for full class name=org.bson.types.ObjectId, simple name=ObjectId"}
[2022-01-19T09:41:36,294][INFO ][logstash.javapipeline ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2022-01-19T09:41:36,760][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:main}
[2022-01-19T09:41:36,795][INFO ][logstash.runner ] Logstash shut down.
And my config file is like this:
input {
jdbc{
jdbc_driver_class => "com.dbschema.MongoJdbcDriver"
jdbc_driver_library => "C:\logstash-7.16.3\logstash-core\lib\jars\mongojdbc3.1.jar"
jdbc_user => ""
jdbc_password => ""
jdbc_connection_string => "jdbc:mongodb://localhost:27017/Owrang"
statement => "db.getCollection('city').find({})"
}
}
filter {
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
action => "index"
index => "city"
hosts => ["localhost:9200"]
}
}
It not works. Please help me. There is no error and I don't know what is happening.