Enterprise Search Mongo-Connector Advanced Rule to Filter by Current Date

Hello! I am trying to ingest records from MongoDB through a Enterprise Search Mongo-Connector. I would like to use an advanced rule to only pull in those where a specific field (expiresAt) is greater than or equal to the current date. The Elastic docs state that the mongodb connector supports aggregation pipelines in the advanced rules, however I am getting errors when trying to set this up.

For example:

  "aggregate": {
    "pipeline": [
        "$addFields": {
          "current_date": { "$toDate": "$$NOW" }
        "$match": {
          "expiresAt": {
            "$gte": "$current_date"

Throws the following error:

[40324:Location40324]: Unrecognized pipeline stage name: '$add_fields' (on server-shard-00-01.o4mny.mongodb.net:27017, modern retry, attempt 1)

Where is it even getting "add_fields" from? Is this not possible?

Hi Colin,

Thank you for your message! May I ask you, which version of connectors/Enterprise Search you're using?

Hi Tim,
After talking to support for awhile, it turned out this was a known issue with 8.7. Now that I have upgraded everything to 8.8, this advanced rule is working. From support:

the reason for the underscore demarcated add_fields is just that we internally convert everything from camel-case to snake-case in our implementation. The actual error you encountered, however, is an issue with our 8.7 release.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.