peledev
December 9, 2021, 10:46am
1
trying to write log from mongodb using jdbc
input {
jdbc {
jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/mongodb-driver-core-3.4.2.jar"
jdbc_driver_class => "Java::mongodb.jdbc.MongoDriver"
jdbc_connection_string => "jdbc:mongodb://<user><password>@<serverip>:27017/stg?authSource=admin"
jdbc_user => "<user>"
jdbc_password => "<password>"
#schedule => "*/10 * * * * *"
statement => "db.logEvents.find({});"
}
}
output {
file {
path => "/var/log/logstash/carobd-mongo.log"
}
}
and i get an error in logstash log :
[2021-12-09T12:35:48,974][DEBUG][logstash.inputs.jdbc ] loading /usr/share/logstash/logstash-core/lib/jars/mongodb-driver-core-3.4.2.jar
Pipeline_id:mongo
Plugin: <LogStash::Inputs::Jdbc jdbc_user=>"user", jdbc_password=><password>, statement=>"db.logEvents.find({});", jdbc_driver_library=>"/usr/share/logstash/logstash-core/lib/jars/mongodb-driver-core-3.4.2.jar", jdbc_connection_string=>"jdbc:mongodb://user:*****@<serverip>:27017/stg?authSource=admin", id=>"d978c12aa530ca4c0d00a4d513b141d959d66a05d400583b2e6046568d83c8b3", jdbc_driver_class=>"Java::mongodb.jdbc.MongoDriver", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_a16f09fa-dc00-4e9b-9c72-6da4159206d7", enable_metric=>true, charset=>"UTF-8">, jdbc_paging_enabled=>false, jdbc_page_size=>100000, jdbc_validate_connection=>false, jdbc_validation_timeout=>3600, jdbc_pool_timeout=>5, sql_log_level=>"info", connection_retry_attempts=>1, connection_retry_attempts_wait_time=>0.5, plugin_timezone=>"utc", last_run_metadata_path=>"/usr/share/logstash/.logstash_jdbc_last_run", use_column_value=>false, tracking_column_type=>"numeric", clean_run=>false, record_last_run=>true, lowercase_column_names=>true, use_prepared_statements=>false>
Error: Java::mongodb.jdbc.MongoDriver not loaded. Are you sure you've included the correct jdbc driver in :jdbc_driver_library?
AquaX
December 9, 2021, 4:04pm
2
I don't think this is the correct jdbc driver
Try this one instead:
JDBC Driver class name: mongodb.jdbc.MongoDriver
URL format: jdbc:mongodb://<serverName>/<databaseName>
Also check out this community project:
peledev
December 13, 2021, 7:50am
3
i did like you said
input {
jdbc {
jdbc_driver_library => "/usr/share/logstash/mongo/mongodb_unityjdbc_full.jar"
jdbc_driver_class => "Java::mongodb.jdbc.MongoDriver"
jdbc_connection_string => "jdbc:mongodb://<server>:27017/stg"
jdbc_user => "<user>"
jdbc_password => "<pass>"
#schedule => "*/10 * * * * *"
statement => "db.logEvents.find({});"
}
}
output {
file {
path => "/var/log/logstash/mongo.log"
}
}
and i get this error now:
<user><server> logstash]$ tail -100 logstash-plain.log | grep -i error
[2021-12-13T09:46:25,317][ERROR][logstash.pipeline ] A plugin had an unrecoverable error. Will restart this plugin.
Error: unable to load /usr/share/logstash/mongojar/mongodb_unityjdbc_full.jar from :jdbc_driver_library, no such file to load -- /usr/share/logstash/mongojar/mongodb_unityjdbc_full
Exception: LogStash::PluginLoadingError
[2021-12-13T09:46:26,332][ERROR][logstash.pipeline ] A plugin had an unrecoverable error. Will restart this plugin.
Error: unable to load /usr/share/logstash/mongojar/mongodb_unityjdbc_full.jar from :jdbc_driver_library, no such file to load -- /usr/share/logstash/mongojar/mongodb_unityjdbc_full
Exception: LogStash::PluginLoadingError
AquaX
December 13, 2021, 2:49pm
5
It says “no such file to load” so double check your path and permissions
peledev
December 15, 2021, 8:03am
6
i made sure it is located in the path
[2021-12-15T10:02:15,283][DEBUG][logstash.agent ] Converging pipelines state {:actions_count=>0}
[2021-12-15T10:02:15,984][DEBUG][logstash.inputs.jdbc ] loading /usr/share/logstash/mongo/mongodb_unityjdbc_full.jar
[2021-12-15T10:02:15,985][ERROR][logstash.pipeline ] A plugin had an unrecoverable error. Will restart this plugin.
Pipeline_id:mongodb
Plugin: <LogStash::Inputs::Jdbc jdbc_user=>"<user>", jdbc_password=><password>, statement=>"db.logEvents.find({});", jdbc_driver_library=>"/usr/share/logstash/mongo/mongodb_unityjdbc_full.jar", jdbc_connection_string=>"jdbc:mongodb://<server>:27017/stg", id=>"b26a801e0007f535ba829b03574fb36cc001fdac19bb6a42c11d81b54fd25ffe", jdbc_driver_class=>"mongodb.jdbc.MongoDriver", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_50c0bf8d-3ce0-4fe4-b5e9-74f5cadf30f1", enable_metric=>true, charset=>"UTF-8">, jdbc_paging_enabled=>false, jdbc_page_size=>100000, jdbc_validate_connection=>false, jdbc_validation_timeout=>3600, jdbc_pool_timeout=>5, sql_log_level=>"info", connection_retry_attempts=>1, connection_retry_attempts_wait_time=>0.5, plugin_timezone=>"utc", last_run_metadata_path=>"/usr/share/logstash/.logstash_jdbc_last_run", use_column_value=>false, tracking_column_type=>"numeric", clean_run=>false, record_last_run=>true, lowercase_column_names=>true, use_prepared_statements=>false>
Error: mongodb.jdbc.MongoDriver not loaded. Are you sure you've included the correct jdbc driver in :jdbc_driver_library?
Exception: LogStash::PluginLoadingError
Stack: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.19/lib/logstash/plugin_mixins/jdbc/jdbc.rb:180:in `open_jdbc_connection'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.19/lib/logstash/plugin_mixins/jdbc/jdbc.rb:243:in `execute_statement'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.19/lib/logstash/inputs/jdbc.rb:309:in `execute_query'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.19/lib/logstash/inputs/jdbc.rb:281:in `run'
/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:426:in `inputworker'
/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:420:in `block in start_input'
[2021-12-15T10:02:16,160][DEBUG][logstash.outputs.file ] Starting flush cycle
see location exist:
[clau@server mongo]$ ls
2021-10-26.zip
drivers
initsources.bat
lib
mongodb_unityjdbc_full.jar
ruby-3.0.3
splunk_unityjdbc_full.jar
unity.ico
unityjdbc.jar
code
icon.png
initsources.sh
logstash-input-mongodb-0.3.0.gem
README.txt
sampleDB
Uninstaller
UnityJDBCDoc.pdf
UnityJDBC_Trial_Install.jar
[clau@server mongo]$ pwd
/usr/share/logstash/mongo
[clau@server mongo]$
system
January 12, 2022, 8:04am
7
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.