Sync from mongoDB to elasticsearch

[deamnguyen (Deam)]

Hi all, I have questions about sync data from mongodb to elasticsearch, please give me permission:

1. There are many ways to sync, which one have i used?

• Logstash, JDBC input plugin

=> Logstash file my config:

jdbc {
jdbc_driver_library => "mongojdbc1.8.jar"
    jdbc_driver_class => "com.dbschema.MongoJdbcDriver"
    jdbc_connection_string => "jdbc: mongodb://localhost:27017/tinh_huyen_xa "
    jdbc_user => ""
    jdbc_password => ""
    jdbc_paging_enabled => true
    schedule => "*/50 * * * * *"
    statement => 
    "
        db.ward.find({})
    "
  }

2. Error:

• I had a really stupid problem

Only SELECT statements allowed with executeQuery()
<I think, the logstash not allow run mongo code>
3. Question:

• The statement above I have omitted a lot (i need to handle a lot of nature). My question is how do i run the mongo code in logstash or do you have any suggestions?

Every answer is appreciated.

[rameshkr1994 (HadoopHelp)]

Hi @deamnguyen welcome .

just take this connector from below link- and apply the same as you are doing-

yougov/mongo-connector

i did the same with sql server-

input {
  jdbc {
    jdbc_driver_library => "C:\Users\Ramesh.kumar\Downloads\Elasticsearch\sqljdbc_4.2\enu\jre8\sqljdbc42.jar"
    #jdbc_driver_class => "com.microsoft.jdbc.sqlserver.SQLServerDriver"
	jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
    jdbc_connection_string => "zzzzzzzzzzzzzzzzzzzzzzzzzzz"
	#jdbc:sqlserver://databaseserver\instance;databaseName=test;integratedSecurity=true
    
    jdbc_user => "dser"
	jdbc_password => "dser"
	jdbc_validate_connection => true
	tracking_column => "dateinserted"
	 use_column_value =>true

    statement => "SELECT * FROM COURSE where dateinserted >:sql_last_value;"
	last_run_metadata_path => "C:\Users\Ramesh.kumar\.logstash_jdbc_last_run"
	#schedule => "* * * * * *"
  }
}

output {
  elasticsearch {
    document_id => "%{id}"
	document_type => "_doc"
    index => "dbdataindex_testauto"
	hosts => ["localhost:9200"]
	}
	stdout{
  codec => rubydebug
	
  }
}

Note-just replace lib. for mongo db and driver /user and pass.

Thanks
HadoopHelp

[deamnguyen (Deam)]

sorry for my delay, I'm grateful for your two option.