# MongoDB-Elasticsearch Integration Via Logstash **URL:** \ **Category:** Logstash\ **Created:** [June 30, 2020, 1:34pm UTC](https://discuss.elastic.co/t/mongodb-elasticsearch-integration-via-logstash/239316 "2020-06-30T13:34:44Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![leoper](https://avatars.discourse-cdn.com/v4/letter/l/ce7236/32.png) [@leoper](https://discuss.elastic.co/u/leoper)\ **Post date:** [June 30, 2020, 1:34pm UTC](https://discuss.elastic.co/t/mongodb-elasticsearch-integration-via-logstash/239316/1 "2020-06-30T13:34:44Z")
Hi everyone. I am trying to make an integration between MongoDB (Version 4.2.7) and Elasticsearch (Version 7.7.0) via Logstash (Version 7.8.0). The goal is to synchronize data from MongoDB to Elasticsearch. The scenario is as follows: 1. An instance of MongoDB Server on 127.0.0.1:2717; 2. Installation of Logstash and the plug-in for mongoDB logstash-input-mongodb; 3. Elasticsearch cluster with 3 nodes: 127.0.0.1:9200, 127.0.0.1:9201, 127.0.0.1:9202; Here is the Logstash configuration file: input { mongodb { uri =\> 'mongodb: // username: password @ localhost: 27017 / dbname' placeholder\_db\_dir =\> 'C: \ Logstash-MongoDB' placeholder\_db\_name =\> 'logstash\_mongodb.db' collection =\> 'collectionName' batch\_size =\> 5000 } } filter { } output { Elasticsearch { hosts =\> ["localhost: 9200", "localhost: 9201", "localhost: 9202"] index =\> "collectionName" } } The pipeline starts and soon "dies". Here is a fragment of the log: ```auto [2020-06-30T12:01:30,991][**ERROR**][logstash.javapipeline] [main] **Pipeline aborted due to error** {:pipeline_id=>"main", :exception=>#<**NoMethodError: undefined method `[]' for nil:NilClass**>, :backtrace=> ["C:/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-mongodb-0.4.1/lib/logstash/inputs/mongodb.rb:99:in `init_placeholder'", "C:/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-mongodb-0.4.1/lib/logstash/inputs/mongodb.rb:113:in `get_placeholder'", "C:/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-mongodb-0.4.1/lib/logstash/inputs/mongodb.rb:160:in `block in update_watched_collections'", "org/jruby/RubyArray.java:1809:in `each'", "C:/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-mongodb-0.4.1/lib/logstash/inputs/mongodb.rb:158:in `update_watched_collections'", "C:/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-mongodb-0.4.1/lib/logstash/inputs/mongodb.rb:182:in `register'", "C:/logstash-7.8.0/logstash-core/lib/logstash/java_pipeline.rb:216:in `block in register_plugins'", "org/jruby/RubyArray.java:1809:in `each'", "C:/logstash-7.8.0/logstash-core/lib/logstash/java_pipeline.rb:215:in `register_plugins'", "C:/logstash-7.8.0/logstash-core/lib/logstash/java_pipeline.rb:326:in `start_inputs'", "C:/logstash-7.8.0/logstash-core/lib/logstash/java_pipeline.rb:286: in `start_workers'", "C:/logstash-7.8.0/logstash-core/lib/logstash/java_pipeline.rb:170:in `run'", "C:/logstash-7.8.0/logstash-core/lib/logstash/java_pipeline.rb:125:in `block in start'"], "pipeline.sources"=>["c:/logstash-7.8.0/config/mongodata.conf"], :thread=>"#"} ``` Thank You very much, leoper. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 30, 2020, 4:24pm UTC](https://discuss.elastic.co/t/mongodb-elasticsearch-integration-via-logstash/239316/2 "2020-06-30T16:24:28Z")
It [ran a find for that collection](https://github.com/phutchins/logstash-input-mongodb/blob/097cc9be8bb57754ec4006e174f0cb5cf3ff6b65/lib/logstash/inputs/mongodb.rb#L99) and got back nil. Are you sure the collection exists? ---
**Author:** ![leoper](https://avatars.discourse-cdn.com/v4/letter/l/ce7236/32.png) [@leoper](https://discuss.elastic.co/u/leoper)\ **Post date:** [June 30, 2020, 4:40pm UTC](https://discuss.elastic.co/t/mongodb-elasticsearch-integration-via-logstash/239316/3 "2020-06-30T16:40:41Z")
Hi Badger. Thank You for your reply. I solved debugging **mongodb.rb** file under Logstash installation. The collection exist, but it was empty. I would have expected that pipeline would have runned with no operation performed. The problem now is that pipeline runs but documents (1 while I am writing) are not transferred on Elasticsearch. No erros occur in the Logstash log. Thank You again! leoper [RubyScreenShot|690x273](https://discuss.elastic.co/uploads/short-url/lyum1B5PAgaIskfvEQd9mXbtZQe.png) ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 30, 2020, 4:46pm UTC](https://discuss.elastic.co/t/mongodb-elasticsearch-integration-via-logstash/239316/4 "2020-06-30T16:46:42Z")
The way that placeholder code works, it runs a find to get the first document, then runs a query to find all documents after the first. It cannot retrieve the first document. It is a [known issue](https://github.com/phutchins/logstash-input-mongodb/issues/94). ---
**Author:** ![leoper](https://avatars.discourse-cdn.com/v4/letter/l/ce7236/32.png) [@leoper](https://discuss.elastic.co/u/leoper)\ **Post date:** [June 30, 2020, 5:15pm UTC](https://discuss.elastic.co/t/mongodb-elasticsearch-integration-via-logstash/239316/5 "2020-06-30T17:15:55Z")
Thank You, I didn't know. However by inserting 2 documents, no one is transferred. In the Logstash log, I see only the last inserted document is fetched. It seems, may be I wrong, as if all documents were considered place holder, so no one is transferred. tks, leoper. [LogstashLogScreenShot|690x87](https://discuss.elastic.co/uploads/short-url/omT6MGD5mMRBGiXDVFmQaQdkr4u.png) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 28, 2020, 5:16pm UTC](https://discuss.elastic.co/t/mongodb-elasticsearch-integration-via-logstash/239316/6 "2020-07-28T17:16:00Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.