I have a cloud version of Elastic Enterprise search (7.9.1). I have integrated Confluence connector with it. After 2 weeks, it has just synced ~450 records, whereas there are more than 25k+ records in my space. There is some issue with Confluence connector since 7.9 release. This behaviour wasn't there in earlier versions and sync was pretty fast (in lest than 4 hrs, all my documents had synced earlier). There is probably a bug injected in recent release. I have raised a bug with the team 4 weeks back, but no solution yet. Has anyone faced similar issue with their Workplace Search setup. Any help is much appreciated. Full sync fails after 3hrs and after updating 27k+ items.
Hi @Prateek_Yadav! Thanks for using Elastic Enterprise Search!
We're not currently aware of any Confluence syncing issues introduced between 7.9.0 and 7.9.1. We are aware of a bug where Spaces are removed from our backend even if they haven't been deleted in confluence. Do you know what "type" of documents you are missing there in the Content Summary? Also, can you take a look at your logs and see what caused your Full Sync to fail?
Thanks,
Sean
There ain't any specific type of document which gets filtered out. I have used trial cloud accounts couple of time to configure Jira/Confluence/Github/GDrive. I had done a successful integration on 22nd July (Not sure what latest cloud version was running then, I guess 7.8). But since August (I guess since 7.9), I see that confluence has stopped syncing completely and full sync fails every single time. But the point is, I'm able to reproduce it every single time on the cloud deployment with out of the box setting (no custom setting applied). I would sincerely encourage you to test this in your dev/test environment while I dig for logs. I have deleted and re-created a fresh environment and have been able to reproduce it every time. Let me get back to you with logs.
No error logs as such. Just INFO level logs are registered.
| Sep 22, 2020, 7:20:32 PM UTC | INFO | i1@us-east-1a | [instance-0000000001] SLM retention snapshot cleanup task complete |
|---|---|---|---|
| Sep 22, 2020, 7:20:32 PM UTC | INFO | i1@us-east-1a | [instance-0000000001] snapshots [cloud-snapshot-2020.09.20-hr76epoxqzmc64wvhjm6sw/-tvJLNGKT5CIhyW74uUgWA] deleted |
| Sep 22, 2020, 7:20:32 PM UTC | INFO | i1@us-east-1a | [instance-0000000001] [found-snapshots] Found stale root level blobs [index-580]. Cleaning them up |
| Sep 22, 2020, 7:20:03 PM UTC | INFO | i1@us-east-1a | [instance-0000000001] [found-snapshots] snapshot retention deleting snapshot [cloud-snapshot-2020.09.20-hr76epoxqzmc64wvhjm6sw/-tvJLNGKT5CIhyW74uUgWA] |
| Sep 22, 2020, 7:20:03 PM UTC | INFO | i1@us-east-1a | [instance-0000000001] starting snapshot retention deletion for [1] snapshots |
| Sep 22, 2020, 7:20:03 PM UTC | INFO | i1@us-east-1a | [instance-0000000001] deleting snapshots [cloud-snapshot-2020.09.20-hr76epoxqzmc64wvhjm6sw] from repository [found-snapshots] |
| Sep 22, 2020, 7:20:00 PM UTC | INFO | i1@us-east-1a | [instance-0000000001] starting SLM retention snapshot cleanup task |
Hi @Prateek_Yadav,
That was the first thing I did yesterday when you reported this. However, other than the discrepancy in "Space" types (like I mentioned above) I am not having the same issue you are seeing. Hence the request for more logs.
Thank you, that's helpful to rule out.
Those logs are the Elasticsearch Logs. You need to look at your Enterprise Search Logs. Look for the message Encountered error in Index, which is logged when sync jobs fail. Because we can see in your screenshot that you have failing sync jobs, we know this log will be present, and it will give us some insight into what is causing your failures.
Hi Sean,
Thanks for your reply!
Do you mind navigating me to the steps where I can find Enterprise Search logs on cloud deployment? I don't see an option on UI.
Thanks
Hi @Sean_Story, just an update on the problem.
Since I had created multiple apps for Confluence in my trial exercise, I was thinking maybe I have been rate-limited by atlassian.
So I asked a colleague of mine to create a trial elastic cloud account and spin up a deployment for Workplace Search. He tried it and added Confluence connector. Unfortunately, sync on his account also has error after ~ 200 items.
Now, there can be only 2 things which must have gone wrong.
1.) my confluence instance (as you have already mentioned that your connector is working fine)
2.) Cloud deployment ( as both our deployment is showing similar behaviour).
I'm trying to narrow down the problem. Let me know if you think trying something else can help.
That's my misunderstanding, I hadn't originally realized you were on a cloud deployment. You can't access the Enterprise Search logs from a cloud deployment, but you can contact Elastic Support to ask them to share your logs. While looking into this for you, I discovered that you already have submitted a support request, so let's keep the conversation going through that channel instead of this one. From digging into your logs, we were able to see that Confluence's API is responding with 500 errors like: {"code":500,"message":"Internal Server Error"}, which is causing the jobs to fail.
This is a possibility. I don't know why Confluence is responding with 500 errors for your setup.
I've validated that our cloud deployments can connect to our test instance of Confluence Cloud without issue. I'm leaning towards thinking this is an issue on the Confluence side. Let's continue the discussion in your official Support ticket for wider audience distribution and reliable SLAs. 
Thanks @Sean_Story for responding to each of the points. Since the support ticket wasn't moving ahead, I thought of seeking help on the forum as I'm very excited to rollout workplace search in my organisation 
I'll continue further discussions there.
Thanks again for giving some insight into it. Appreciate it.