I have a Index setup with a MongoDB Connector that is pulling from a collection of ~1.4M records. The initial run was fine and took about 1.5 hours to complete. However, now I'm trying to re-run the sync every day so pull in new records, but when I try to run the sync it fails after about 3 minutes without pulling in any new records. The error I get in the ElasticCloud UI is:
Sync failure: The job has not seen any update for some time..
What does this error mean? I'm interpreting it as meaning the connector tried to find new records but didn't find any after 3 minutes. With such a large collection I'm wondering if the problem is that it's basically timing out after it reviews records in the collection for a few minutes and doesn't find any new ones to add to the index?
For anyone who stumbles upon this issue and is experiencing something similar, Elastic's support responded to me and said they think it might have something to do with the fact that I'm trying to sync > 500k records. Apparently the connector (which is still in beta) doesn't do well with that many records. They suggested I increase my Enterprise Search from 4GB to 8GB of memory, but that also doubles the hourly cost of my deployment so not the best option for me. I tried increasing the Enterprise Search memory temporarily and that did not fix the issue. I will update this if I get any more information from support.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.