# Data migration from mongodb to elasticsearch

**URL:** <https://discuss.elastic.co/t/data-migration-from-mongodb-to-elasticsearch/264311>\
**Category:** Logstash\
**Created:** [February 15, 2021, 12:42pm UTC](https://discuss.elastic.co/t/data-migration-from-mongodb-to-elasticsearch/264311 "2021-02-15T12:42:27Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Pankaj1](https://avatars.discourse-cdn.com/v4/letter/p/8baadc/32.png) [@Pankaj1](https://discuss.elastic.co/u/Pankaj1)\
**Post date:** [February 15, 2021, 12:42pm UTC](https://discuss.elastic.co/t/data-migration-from-mongodb-to-elasticsearch/264311/1 "2021-02-15T12:42:27Z")

</div>

> I am doing data migration from mongodb to elasticsearch by using logstash . I am using jdbc mongodb jar as a plugin .  
> I have aggregation query to get the records from mongodb and insert into elasticsearch .  
> The issues are it is fetching the first record from top in mongod and inserting into elasticsearch and rest document did not insert into elasticsearch . And when i saw the logstash server and i found that the records which has been inserted into elasticsearch it is printing multiple times .
> 
> ## conf file
> 
> input {  
> jdbc{  
> # NOT THIS # jdbc\_driver\_class =\> "Java::mongodb.jdbc.MongoDriver"  
> jdbc\_driver\_class =\> "com.dbschema.MongoJdbcDriver"  
> jdbc\_driver\_library =\> "D:/ELK/jarfile/mongojdbc3.0.jar"  
> jdbc\_user =\> "" #no user and pwd  
> jdbc\_password =\> ""
> 
> # use\_column\_value =\> true
> 
> # tracking\_column =\> "%{[application\_id]}"
> 
> ```
> jdbc_connection_string => "jdbc:mongodb://user
> 
> ```
> 
> > Blockquote
> 
> :pass@mongodb.sandboxforbank.com:27017/Feature-QA-4?ssl=true"
> 
> # jdbc\_connection\_string =\> ""
> 
> ```
> statement => "             
> db.oaoapplicants.aggregate([
> {
> $match: {
> _applicationStatus: 'ACTIVE',
> }
> },
> {
> '$lookup': {
> 'from': 'oaoproductdetails',
> let: {
> product_code: '$product_code'
> },
> pipeline: [
> {
> $match: {
> $expr: {
> $and: [
> {
> $eq: [
> '$product_code',
> '$$product_code'
> ]
> }
> ]
> }
> }
> },
> {
> $project: {
> _id: 0
> }
> }
> ],
> 'as': 'productDetails'
> }
> },
> 
> ```
> 
> {  
> $unwind : {  
> path : '$productDetails',  
> preserveNullAndEmptyArrays: true  
> }  
> },  
> {  
> '$lookup': {  
> 'from': 'applicationtasksconfigs',  
> let: {  
> application\_id: '$application\_id'  
> },  
> pipeline: [  
> {  
> $match: {  
> $expr: {  
> $and: [  
> {  
> $eq: [  
> '$application\_id',  
> '$$application\_id'  
> ]  
> },  
> {  
> $ne: [  
> '$assignedTo.username',  
> 'Unassigned'  
> ]  
> },  
> {  
> $eq: [  
> '$deleted',  
> false  
> ]  
> }  
> ]  
> }  
> }  
> }, {  
> $project: {  
> \_id: 0,  
> 'operationalLevelAgreement.overall.\_id':0,  
> 'operationalLevelAgreement.queued.\_id':0,  
> 'operationalLevelAgreement.assigned.\_id':0,  
> }  
> }  
> ],  
> 'as': 'task'  
> }  
> },  
> {  
> $unwind :{  
> path : '$task',  
> preserveNullAndEmptyArrays: true  
> }
> 
> ```
> },
> {
> $project: {
> '_id': 0,
> 'application_id': '$application_id',
> 'am_accountNo': '$am_accountNo'
> }
> },
> 
> ```
> 
> {  
> $limit : 5  
> }  
> ])  
> "
> 
> }  
> }
> 
> output {  
> stdout {  
> codec =\> rubydebug  
> }  
> elasticsearch {  
> hosts =\> "[http://localhost:9200](http://localhost:9200)"  
> index =\> "abc\_index3"  
> document\_id =\> "%{[document][application\_id]}"  
> }  
> }
> 
> ## output in logstash server
> 
> {  
> "@version" =\> "1",  
> "document" =\> {  
> "am\_accountNo" =\> "846176",  
> "application\_id" =\> "DB2021012131408"  
> },  
> "@timestamp" =\> 2021-02-15T12:40:05.952Z  
> }  
> {  
> "@version" =\> "1",  
> "document" =\> {  
> "am\_accountNo" =\> "846176",  
> "application\_id" =\> "DB2021012131408"  
> },  
> "@timestamp" =\> 2021-02-15T12:40:05.933Z  
> }  
> [2021-02-15T18:10:06,249][INFO][org.mongodb.driver.connection][main][b1604bb1b4a7cd7c375134ae04cfb5db329957ef22629f3eed0409196d19df6a] Closed connection [connectionId{localValue:2, serverValue:24532}] to [mongodb.sandboxforbank.com:27017](http://mongodb.sandboxforbank.com:27017) because the pool has been closed.  
> {  
> "@version" =\> "1",  
> "document" =\> {  
> "am\_accountNo" =\> "846176",  
> "application\_id" =\> "DB2021012131408"  
> },  
> "@timestamp" =\> 2021-02-15T12:40:05.949Z  
> }  
> {  
> "@version" =\> "1",  
> "document" =\> {  
> "am\_accountNo" =\> "846176",  
> "application\_id" =\> "DB2021012131408"  
> },  
> "@timestamp" =\> 2021-02-15T12:40:05.949Z  
> }  
> {  
> "@version" =\> "1",  
> "document" =\> {  
> "am\_accountNo" =\> "846176",  
> "application\_id" =\> "DB2021012131408"  
> },  
> "@timestamp" =\> 2021-02-15T12:40:05.950Z  
> }  
> [2021-02-15T18:10:07,633][INFO][logstash.javapipeline][main] Pipeline terminated {"pipeline.id"=\>"main"}  
> [2021-02-15T18:10:07,824][INFO][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline\_id=\>:main}  
> [2021-02-15T18:10:07,882][INFO][logstash.runner] Logstash shut down.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 15, 2021, 12:42pm UTC](https://discuss.elastic.co/t/data-migration-from-mongodb-to-elasticsearch/264311/2 "2021-03-15T12:42:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.