Hi,
I wanted to ship the data from mongo db to elastic search. In my mongodb collection for some records I replaced the object id with custom id. So mongodb plugin is not working for me. In mongodb i have documents with the default object and my custom id. My question is how can i ship the entire data which are having default object id and custom id's to elasticsearch. I tried JDBC plugin but it only inputs few documents which have custom id not all and it is ignoring the documents with default object id. Below is the configuration file that I use.
input {
jdbc {
jdbc_driver_library =>"C:/Users/xxxx/Downloads/Backup/Downloads/mongo-java-driver-3.6.4.jar,C:/Users/xxxx/Downloads/Backup/Downloads/mongojdbc1.2.jar"
jdbc_driver_class => "com.dbschema.MongoJdbcDriver"
jdbc_connection_string => "jdbc:mongodb://ip:port/DB-NAME"
jdbc_user => "admin"
schedule => "*/10 * * * * * UTC"
statement => "db.getCollection('Collection name').find({ Timestamp: { $gte: (:sql_last_value)}})"
last_run_metadata_path => "C:\Users\xxxx\Downloads\ELK-7.3.2\logstash-7.3.2\logstash-7.3.2\.logstash_jdbc_last_run"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
user => "elastic"
password => "changeme"
index => "mongodocuments"
}
stdout { }
}
Need immediate help. Thanks in advance.