MongoDB-Elasticsearch Integration Via Logstash

Elastic Stack Logstash
1

Hi everyone.

I am trying to make an integration between MongoDB (Version 4.2.7) and Elasticsearch (Version 7.7.0) via Logstash (Version 7.8.0).
The goal is to synchronize data from MongoDB to
Elasticsearch.

The scenario is as follows:

  1. An instance of MongoDB Server on 127.0.0.1:2717;
  2. Installation of Logstash and the plug-in for mongoDB logstash-input-mongodb;
  3. Elasticsearch cluster with 3 nodes: 127.0.0.1:9200, 127.0.0.1:9201, 127.0.0.1:9202;

Here is the Logstash configuration file:

input
{
mongodb
{
uri => 'mongodb: // username: password @ localhost: 27017 / dbname'
placeholder_db_dir => 'C: \ Logstash-MongoDB'
placeholder_db_name => 'logstash_mongodb.db'
collection => 'collectionName'
batch_size => 5000
}
}
filter
{
}
output
{
Elasticsearch
{
hosts => ["localhost: 9200", "localhost: 9201", "localhost: 9202"]
index => "collectionName"
}
}

The pipeline starts and soon "dies". Here is a fragment of the log:

[2020-06-30T12:01:30,991][**ERROR**][logstash.javapipeline    ]
[main] **Pipeline aborted due to error** {:pipeline_id=>"main", :exception=>#<**NoMethodError: undefined method `[]' for nil:NilClass**>, :backtrace=>
["C:/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-mongodb-0.4.1/lib/logstash/inputs/mongodb.rb:99:in `init_placeholder'",
"C:/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-mongodb-0.4.1/lib/logstash/inputs/mongodb.rb:113:in `get_placeholder'",
"C:/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-mongodb-0.4.1/lib/logstash/inputs/mongodb.rb:160:in `block in update_watched_collections'",
"org/jruby/RubyArray.java:1809:in `each'", 
"C:/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-mongodb-0.4.1/lib/logstash/inputs/mongodb.rb:158:in `update_watched_collections'", 
"C:/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-mongodb-0.4.1/lib/logstash/inputs/mongodb.rb:182:in `register'", 
"C:/logstash-7.8.0/logstash-core/lib/logstash/java_pipeline.rb:216:in `block in register_plugins'", 
"org/jruby/RubyArray.java:1809:in `each'", "C:/logstash-7.8.0/logstash-core/lib/logstash/java_pipeline.rb:215:in `register_plugins'", 
"C:/logstash-7.8.0/logstash-core/lib/logstash/java_pipeline.rb:326:in `start_inputs'", "C:/logstash-7.8.0/logstash-core/lib/logstash/java_pipeline.rb:286:
in `start_workers'", "C:/logstash-7.8.0/logstash-core/lib/logstash/java_pipeline.rb:170:in `run'", 
"C:/logstash-7.8.0/logstash-core/lib/logstash/java_pipeline.rb:125:in `block in start'"], 
"pipeline.sources"=>["c:/logstash-7.8.0/config/mongodata.conf"], :thread=>"#<Thread:0x55e4e5c4 run>"}

Thank You very much, leoper.

2

It ran a find for that collection and got back nil. Are you sure the collection exists?

3

Hi Badger.

Thank You for your reply.

I solved debugging mongodb.rb file under Logstash installation. The collection exist, but it was empty. I would have expected that pipeline would have runned with no operation performed.

The problem now is that pipeline runs but documents (1 while I am writing) are not transferred on Elasticsearch. No erros occur in the Logstash log.

Thank You again!
leoper

RubyScreenShot|690x273

4

The way that placeholder code works, it runs a find to get the first document, then runs a query to find all documents after the first. It cannot retrieve the first document. It is a known issue.

5

Thank You, I didn't know.

However by inserting 2 documents, no one is transferred. In the Logstash log, I see only the last inserted document is fetched.
It seems, may be I wrong, as if all documents were considered place holder, so no one is transferred.

tks, leoper.

LogstashLogScreenShot|690x87

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.