Enterprise Search Mongo-Connector Advanced Rule to Filter by Current Date

[pocketcolin (Colin Ulin)]

Hello! I am trying to ingest records from MongoDB through a Enterprise Search Mongo-Connector. I would like to use an advanced rule to only pull in those where a specific field (expiresAt) is greater than or equal to the current date. The Elastic docs state that the mongodb connector supports aggregation pipelines in the advanced rules, however I am getting errors when trying to set this up.

For example:

{
  "aggregate": {
    "pipeline": [
      {
        "$addFields": {
          "current_date": { "$toDate": "$$NOW" }
        }
      },
      {
        "$match": {
          "expiresAt": {
            "$gte": "$current_date"
          }
        }
      }
    ]
  }
}

Throws the following error:

[40324:Location40324]: Unrecognized pipeline stage name: '$add_fields' (on server-shard-00-01.o4mny.mongodb.net:27017, modern retry, attempt 1)

Where is it even getting "add_fields" from? Is this not possible?

[Tim_Grein]

Hi Colin,

Thank you for your message! May I ask you, which version of connectors/Enterprise Search you're using?

[pocketcolin (Colin Ulin)]

Hi Tim,
After talking to support for awhile, it turned out this was a known issue with 8.7. Now that I have upgraded everything to 8.8, this advanced rule is working. From support:

the reason for the underscore demarcated add_fields is just that we internally convert everything from camel-case to snake-case in our implementation. The actual error you encountered, however, is an issue with our 8.7 release.