Dropbox connector doesn't pickup sync jobs

Elastic Search
,
1

We are using workspace search for searching thru dropbox documents. These documents are part of a business team folder and accessable for the users. In workspace search it works fine. But since version 9.0 workspace search and elasticsearch hosted connectors are no longer supported. So before (now 8.18) upgrading to 9.0 we want to install a self hosted docker dropbox connetor connected to a new index.
We followed all the steps installing the connector and connect it to the dropbox application. So far all signs are green. Connection to the elk cloud can be made as the connection to the dropbox application. We manually started a full sync but nothing is happening. No actions visible in the logging. We turned on debugging mode for the connector and still nothing can be seen. We tried to turn on a scheduled full sync, but still no action.
I started a case at the supportdesk but no real help from them (over 5 days between a responce and no real answers to my questions).

Does anyone have a clue where to find out what is going on?? All help is appreciated

Startup info:

docker run -v "$HOME/elastic-connectors:/config" --tty --rm docker.elastic.co/integrations/elastic-connectors:8.18.0 /app/bin/elastic-ingest -c /config/config3.yml

[FMWK][08:16:57][INFO] Running connector service version 8.18.0

[FMWK][08:16:57][INFO] Loading config from /config/config3.yml

[FMWK][08:16:57][INFO] Running preflight checks

[FMWK][08:16:57][INFO] Waiting for Elasticsearch at https://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.gcp.elastic-cloud.com:443 (so far: 0 secs)

[FMWK][08:16:57][INFO] Elasticsearch 8.18.0 and Connectors 8.18.0 are compatible

[FMWK][08:16:57][INFO] Extraction service is not configured, skipping its preflight check.

[FMWK][08:16:57][INFO] [job_scheduling_service] Job Scheduling Service started, listening to events from https://xxxxxxxxxxxxxxxxxxxxxxxx.gcp.elastic-cloud.com:443

[FMWK][08:16:57][INFO] [content_sync_job_execution_service] Content sync job execution service started, listening to events from https://xxxxxxxxxxxxxxxxxxxxxxxx.gcp.elastic-cloud.com:443

[FMWK][08:16:58][INFO] [access_control_sync_job_execution_service] Access control sync job execution service started, listening to events from https://xxxxxxxxxxxxxxxxxxxxxxxxxxxxx.gcp.elastic-cloud.com:443

[FMWK][08:16:58][INFO] [Connector id: XXXXXXXXXXXXXXXXXX, index name: connector-dropbox-a58b] Successfully connected to Dropbox

[FMWK][08:16:58][INFO] [job_scheduling_service] Connector is configured correctly and can reach the data source

[FMWK][08:17:29][INFO] [Connector id: XXXXXXXXXXXXXXXXXXXXX, index name: connector-dropbox-a58b] Successfully connected to Dropbox

some debugging:
[FMWK][08:08:09][INFO] [Connector id: XXXXXXXXXXXXXXXXXX index name: connector-dropbox-a58b] Successfully connected to Dropbox

[FMWK][08:08:09][DEBUG] [Connector id: XXXXXXXXXXXXXXXXXX index name: connector-dropbox-a58b] Filtering is in state valid, skipping...

[FMWK][08:08:09][INFO] [job_scheduling_service] Connector is configured correctly and can reach the data source

[FMWK][08:08:09][DEBUG] Cancelling sleeps...

[FMWK][08:08:09][DEBUG] [job_scheduling_service] Scheduler woke up at 2025-05-20 08:08:09.130397+00:00. Previously woke up at 2025-05-20 08:07:38.526543+00:00.

[FMWK][08:08:09][DEBUG] [job_scheduling_service] Last sync was scheduled at None

[FMWK][08:08:09][DEBUG] [Connector id: XXXXXXXXXXXXXXXXXX index name: connector-dropbox-a58b] Next 'access_control' sync is at None

[FMWK][08:08:09][DEBUG] [Connector id: XXXXXXXXXXXXXXXXXX index name: connector-dropbox-a58b] 'access_control' sync scheduling is disabled

[FMWK][08:08:09][DEBUG] [job_scheduling_service] Scheduler woke up at 2025-05-20 08:08:09.138141+00:00. Previously woke up at 2025-05-20 08:07:38.526543+00:00.

[FMWK][08:08:09][DEBUG] [job_scheduling_service] Last sync was scheduled at None

[FMWK][08:08:09][DEBUG] [Connector id: XXXXXXXXXXXXXXXXXX index name: connector-dropbox-a58b] Next 'incremental' sync is at None

[FMWK][08:08:09][DEBUG] [Connector id: XXXXXXXXXXXXXXXXXX index name: connector-dropbox-a58b] 'incremental' sync scheduling is disabled

[FMWK][08:08:09][DEBUG] [job_scheduling_service] Scheduler woke up at 2025-05-20 08:08:09.145961+00:00. Previously woke up at 2025-05-20 08:07:38.526543+00:00.

[FMWK][08:08:09][DEBUG] [job_scheduling_service] Last sync was scheduled at 2025-05-20 08:00:00+00:00

[FMWK][08:08:09][DEBUG] [Connector id: XXXXXXXXXXXXXXXXXX index name: connector-dropbox-a58b] Next 'full' sync is at 2025-05-20 08:10:00+00:00

[FMWK][08:08:09][DEBUG] [Connector id: XXXXXXXXXXXXXXXXXX index name: connector-dropbox-a58b] Next 'full' sync due in 110 seconds

[FMWK][08:08:19][DEBUG] [access_control_sync_job_execution_service] Polling every 30 seconds for access control sync job execution

[FMWK][08:08:19][DEBUG] [content_sync_job_execution_service] Polling every 30 seconds for content sync job execution

[FMWK][08:08:39][DEBUG] [job_scheduling_service] Polling every 30 seconds for Job Scheduling

[FMWK][08:08:39][DEBUG] [Connector id: XXXXXXXXXXXXXXXXXX index name: connector-dropbox-a58b] Status is Status.CONNECTED

[FMWK][08:08:39][DEBUG] [Connector id: XXXXXXXXXXXXXXXXXX index name: connector-dropbox-a58b] Validating configuration

[FMWK][08:08:39][DEBUG] [Connector id: XXXXXXXXXXXXXXXXXX index name: connector-dropbox-a58b] Pinging the backend

[FMWK][08:08:39][DEBUG] [Connector id: XXXXXXXXXXXXXXXXXX index name: connector-dropbox-a58b] Access Token generated successfully

[FMWK][08:08:39][DEBUG] [Connector id: XXXXXXXXXXXXXXXXXX index name: connector-dropbox-a58b] Generating aiohttp client session

[FMWK][08:08:39][INFO] [Connector id: XXXXXXXXXXXXXXXXXX index name: connector-dropbox-a58b] Successfully connected to Dropbox

[FMWK][08:08:39][DEBUG] [Connector id: XXXXXXXXXXXXXXXXXX index name: connector-dropbox-a58b] Filtering is in state valid, skipping...

[FMWK][08:08:39][INFO] [job_scheduling_service] Connector is configured correctly and can reach the data source

[FMWK][08:08:39][DEBUG] Cancelling sleeps...

[FMWK][08:08:39][DEBUG] [job_scheduling_service] Scheduler woke up at 2025-05-20 08:08:39.777413+00:00. Previously woke up at 2025-05-20 08:08:09.154808+00:00.

[FMWK][08:08:39][DEBUG] [job_scheduling_service] Last sync was scheduled at None

[FMWK][08:08:39][DEBUG] [Connector id: XXXXXXXXXXXXXXXXXX index name: connector-dropbox-a58b] Next 'access_control' sync is at None

[FMWK][08:08:39][DEBUG] [Connector id: XXXXXXXXXXXXXXXXXX index name: connector-dropbox-a58b] 'access_control' sync scheduling is disabled

[FMWK][08:08:39][DEBUG] [job_scheduling_service] Scheduler woke up at 2025-05-20 08:08:39.792104+00:00. Previously woke up at 2025-05-20 08:08:09.154808+00:00.

[FMWK][08:08:39][DEBUG] [job_scheduling_service] Last sync was scheduled at None

[FMWK][08:08:39][DEBUG] [Connector id: XXXXXXXXXXXXXXXXXX index name: connector-dropbox-a58b] Next 'incremental' sync is at None

[FMWK][08:08:39][DEBUG] [Connector id: XXXXXXXXXXXXXXXXXX index name: connector-dropbox-a58b] 'incremental' sync scheduling is disabled

[FMWK][08:08:39][DEBUG] [job_scheduling_service] Scheduler woke up at 2025-05-20 08:08:39.804151+00:00. Previously woke up at 2025-05-20 08:08:09.154808+00:00.

[FMWK][08:08:39][DEBUG] [job_scheduling_service] Last sync was scheduled at 2025-05-20 08:00:00+00:00

[FMWK][08:08:39][DEBUG] [Connector id: XXXXXXXXXXXXXXXXXX index name: connector-dropbox-a58b] Next 'full' sync is at 2025-05-20 08:10:00+00:00

[FMWK][08:08:39][DEBUG] [Connector id: XXXXXXXXXXXXXXXXXX index name: connector-dropbox-a58b] Next 'full' sync due in 80 seconds

[FMWK][08:08:49][DEBUG] [access_control_sync_job_execution_service] Polling every 30 seconds for access control sync job execution

[FMWK][08:08:49][DEBUG] [content_sync_job_execution_service] Polling every 30 seconds for content sync job execution
2

Hey @hansko,

Unfortunately the logs don't show anything special - I could see that the framework saw your schedule and a content sync job should have been scheduled at ~ 08:10:00 UTC.

There is nothing indicating that there has been any job execution.

Have it been scheduled? Do you see it in UI?
Do you also see jobs scheduled when you schedule sync manually?

Here's an example of how it looks for me when I do testing:

image
image2566×524 39.9 KB

3

That is the problem.

When i do "get _connector/_sync_job" the result is a scheduled sync jobs with status pending.

When I look at the connector overview, there are no sync jobs visible. This is the same in the logging. They don't show any sync job activity. When i schedule full sync jobs every hour. After 24 hours you see 24 pending sync_jobs but 0 documents and no jobs in the overview. If there is some misconfiguration I expect error in the logging or at least some warnings. Nothing!!

I also tried to connect from within the docker to the ELK cloud using the used API key. It is working. Using the the api-user/ api-secret and dropbox refresh key connecting the user and folders in dropbox. No errors and all is visible. So I'am puzzeld what is going on here.

4

Have you upgraded from a different version of stack?

We've had similar problem happening when mappings for .elastic-connectors-sync-jobs-v1 index were corrupted due to a problematic migration.

If you don't have any jobs that you want to keep information about, dropping .elastic-connectors-sync-jobs-v1 index and scheduling a sync manually can resolve the problem.

Can you give it a try, if losing all info about sync jobs is acceptable to you?

If not, posting mappings of .elastic-connectors-sync-jobs-v1 could help us debug the problem.

5

We upgraded to version 8.18 ( cloud version ). Before upgrading to version 9.0 we would like to implement the new connectors before we shutdown the workplace search. So there are no sync-job and no older converted sync_jobs. Scheduled sync_jobs are deleted daily by me so I can keep track of what is going on.

6

I see. Upgrade might have caused it even if you haven't ran any connectors before the upgrade.

Can you try deleting the index with DELETE .elastic-connectors-sync-jobs-v1 and run a full sync manually for this connector?

7

I think this was the problem!! Deleting this index and manualy starting a full sync.
In the overview now a sync_job is visible and running. Also the number of documents are rising. Thanks !!

After the full sync i will inspect the data. For now I will let it run.

1 Like