Sync from mongoDB to elasticsearch

Elastic Stack Elasticsearch
1

Hi all, I have questions about sync data from mongodb to elasticsearch, please give me permission:

  1. There are many ways to sync, which one have i used?

=> Logstash file my config:

jdbc {
jdbc_driver_library => "mongojdbc1.8.jar"
    jdbc_driver_class => "com.dbschema.MongoJdbcDriver"
    jdbc_connection_string => "jdbc: mongodb://localhost:27017/tinh_huyen_xa "
    jdbc_user => ""
    jdbc_password => ""
    jdbc_paging_enabled => true
    schedule => "*/50 * * * * *"
    statement => 
    "
        db.ward.find({})
    "
  }
  1. Error:
  • I had a really stupid problem

Only SELECT statements allowed with executeQuery()
<I think, the logstash not allow run mongo code>
3. Question:

  • The statement above I have omitted a lot (i need to handle a lot of nature). My question is how do i run the mongo code in logstash or do you have any suggestions?

Every answer is appreciated.

2

Hi @deamnguyen welcome :slight_smile: .

just take this connector from below link- and apply the same as you are doing-

i did the same with sql server-

input {
  jdbc {
    jdbc_driver_library => "C:\Users\Ramesh.kumar\Downloads\Elasticsearch\sqljdbc_4.2\enu\jre8\sqljdbc42.jar"
    #jdbc_driver_class => "com.microsoft.jdbc.sqlserver.SQLServerDriver"
	jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
    jdbc_connection_string => "zzzzzzzzzzzzzzzzzzzzzzzzzzz"
	#jdbc:sqlserver://databaseserver\instance;databaseName=test;integratedSecurity=true
    
    jdbc_user => "dser"
	jdbc_password => "dser"
	jdbc_validate_connection => true
	tracking_column => "dateinserted"
	 use_column_value =>true

    statement => "SELECT * FROM COURSE where dateinserted >:sql_last_value;"
	last_run_metadata_path => "C:\Users\Ramesh.kumar\.logstash_jdbc_last_run"
	#schedule => "* * * * * *"
  }
}

output {
  elasticsearch {
    document_id => "%{id}"
	document_type => "_doc"
    index => "dbdataindex_testauto"
	hosts => ["localhost:9200"]
	}
	stdout{
  codec => rubydebug
	
  }
}

Note-just replace lib. for mongo db and driver /user and pass.

Thanks
HadoopHelp

1 Like
3

sorry for my delay, I'm grateful for your two option.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.