MongoDB input plug-in for logstash

Elastic Stack Logstash
1

Hi Team,

I am using Logstash (v8.17.1) to sync data from MongoDB to Elasticsearch. The document creation works fine, but I am facing issues with syncing only updated documents from MongoDB.

I tried filtering documents using,

filter=> { "last_updated": { "$gte": { "$date": "{{placeholder}}" } } }

but Logstash throws an error:

[ERROR][logstash.inputs.mongodb] Unknown setting 'filter' for mongodb

Tried using a MongoDB aggregation pipeline:

input {
  mongodb {
    uri => "mongodb://localhost:27017/Resumes"
    collection => "CoarthaTechnoSolutions"
    placeholder_db_dir => "/home/anand/logstash-mongo/"
    placeholder_db_name => "logstash_sqlite.db"
    pipeline => '[{ "$match": { "last_updated": { "$gte": { "$date": "{{placeholder}}" } } } }]'
  }
}

Suggest me a best way,

My Questions:

  1. What is the recommended way to fetch only recently updated documents from MongoDB in Logstash?
  2. Is pipeline officially supported in MongoDB input plugin?
  3. Would Change Streams be a better approach? If yes, how can I integrate them with Logstash?

Thanks