Logstash-output-mongodb plugin for Azure cosmos DB

Elastic Stack Logstash
1

I have an Azure cosmos DB with MongoDB API support, its connection string is in the format of mongodb://username:password@URL.

Logstash can push data to this DB using the community plugin logstash-output-documentdb but it stores the data as JSON. Instead I want to store the data as BSON. So I tried the official logstash-output-mongodb and Iam getting Not Authenticated & Unauthorized error as below.

[2018-09-26T02:42:25,677][WARN ][logstash.outputs.mongodb ] Failed to send event to MongoDB {:event=>#<LogStash::Event:0x19f239aa>, :exception=>#<Mongo::Auth::Unauthorized: User mymongodbuser is not authorized to access .>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/mongo-2.0.6/lib/mongo/auth/cr/conversation.rb:114:in `validate!'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/mongo-2.0.6/lib/mongo/auth/cr/conversation.rb:57:in `continue'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/mongo-2.0.6/lib/mongo/auth/cr.rb:58:in `login'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/mongo-2.0.6/lib/mongo/server/connection.rb:64:in `connect!'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/mongo-2.0.6/lib/mongo/server/connectable.rb:93:in `ensure_connected'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/mongo-2.0.6/lib/mongo/server/connection.rb:130:in `write'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/mongo-2.0.6/lib/mongo/server/connectable.rb:66:in `block in dispatch'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/mongo-2.0.6/lib/mongo/loggable.rb:44:in `log'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/mongo-2.0.6/lib/mongo/loggable.rb:67:in `log_debug'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/mongo-2.0.6/lib/mongo/server/connectable.rb:65:in `dispatch'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/mongo-2.0.6/lib/mongo/operation/executable.rb:35:in `block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/mongo-2.0.6/lib/mongo/server/connection_pool.rb:99:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/mongo-2.0.6/lib/mongo/server/context.rb:63:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/mongo-2.0.6/lib/mongo/operation/executable.rb:34:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/mongo-2.0.6/lib/mongo/operation/write/insert.rb:72:in `execute_write_command'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/mongo-2.0.6/lib/mongo/operation/write/insert.rb:62:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/mongo-2.0.6/lib/mongo/collection.rb:190:in `insert_many'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/mongo-2.0.6/lib/mongo/collection.rb:175:in `insert_one'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-mongodb-3.1.4/lib/logstash/outputs/mongodb.rb:100:in `receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:89:in `block in multi_receive'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:89:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:114:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:97:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:373:in `block in output_batch'", "org/jruby/RubyHash.java:1343:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:372:in `output_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:324:in `worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:286:in `block in start_workers'"]}
[2018-09-26T02:42:28,791][WARN ][logstash.outputs.mongodb ] Failed to send event to MongoDB {:event=>#<LogStash::Event:0x19f239aa>, :exception=>#<Mongo::Error::OperationFailure: Not Authenticated (13), Not Authenticated (13)>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/mongo-2.0.6/lib/mongo/operation/result.rb:214:in `validate!'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/mongo-2.0.6/lib/mongo/operation/write/insert.rb:72:in `execute_write_command'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/mongo-2.0.6/lib/mongo/operation/write/insert.rb:62:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/mongo-2.0.6/lib/mongo/collection.rb:190:in `insert_many'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/mongo-2.0.6/lib/mongo/collection.rb:175:in `insert_one'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-mongodb-3.1.4/lib/logstash/outputs/mongodb.rb:100:in `receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:89:in `block in multi_receive'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:89:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:114:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:97:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:373:in `block in output_batch'", "org/jruby/RubyHash.java:1343:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:372:in `output_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:324:in `worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:286:in `block in start_workers'"]}

Logstash output config is

    mongodb {
       id => "mymongodbuser"
       collection => "MyCollection"
       database => "MyMongoDB"
       uri => "mongodb://mymongodbuser:PASSWORD
@mymongodb.documents.azure.com:10255/?ssl=true&replicaSet=globaldb"
      codec => "json"
    }

Anyone have pushed data from logstash to Azure cosmos DB in BSON format ?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.