Unknown problem occurred when syncing

Elastic Search
1

I added GoogleDrive as a source for WorkplaceSearch.
After a few hours, I was able to retrieve the data from GoogleDrive and search for documents from WorkplaceSearch, but suddenly I got the following message.

Unknown problem occurred when syncing. [Check diagnostic bundle for details - Message id: 60d02e4984c212b38a33d4cd].

How do I resolve this?

I'm posting the contents of a json file that I got by pressing the Download diagnostics data button in Content Sources > Google Drive > Source Settings.

{
  "general": {
    "version": "7.13.2",
    "revision": "af74d68aafaab071eafa00c2ab628ecf5a1daa27",
    "elasticsearchVersion": "7.11.0"
  },
  "source": {
    "id": "60c09c5b84c21256a3a17b64",
    "serviceType": "google_drive",
    "name": "Google Drive",
    "context": "organization",
    "dlpEnabled": false,
    "recentJobs": {
      "incremental": [
        {
          "createdAt": "2021-06-21T06:14:28Z",
          "status": "error",
          "fatalException": {
            "friendly_message": "Unknown problem occurred when syncing.",
            "stack_trace": "/usr/share/enterprise-search/lib/war/frito_pie/lib/secret_keeper/google.class:35:in `refresh_client\u0027: undefined method `[]\u0027 for nil:NilClass\n\tfrom /usr/share/enterprise-search/lib/war/frito_pie/lib/secret_keeper/google.class:19:in `refresh\u0027\n\tfrom /usr/share/enterprise-search/lib/war/frito_pie/lib/secret_keeper.class:274:in `ensure_valid!\u0027\n\tfrom /usr/share/enterprise-search/lib/war/frito_pie/lib/secret_keeper.class:249:in `fetch_secret_details\u0027\n\tfrom /usr/share/enterprise-search/lib/war/frito_pie/lib/secret_keeper.class:115:in `fetch_content_source_secret_details!\u0027\n\tfrom /usr/share/enterprise-search/lib/war/frito_pie/app/models/frito_pie/content_source.class:663:in `authorization_details\u0027\n\tfrom /usr/share/enterprise-search/lib/war/frito_pie/app/models/frito_pie/content_source.class:671:in `authorization_data\u0027\n\tfrom /usr/share/enterprise-search/lib/war/frito_pie/lib/frito_pie/content_source_service/google_drive.class:141:in `block in web_client\u0027\n\tfrom org/jruby/RubyKernel.java:1897:in `tap\u0027\n\tfrom /usr/share/enterprise-search/lib/war/frito_pie/lib/frito_pie/content_source_service/google_drive.class:140:in `web_client\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/content_sources/base/extractor.class:93:in `client\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/content_sources/google_drive/extractor.class:193:in `get_files\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/content_sources/google_drive/extractor.class:123:in `block in yield_files\u0027\n\tfrom org/jruby/RubyKernel.java:1442:in `loop\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/content_sources/google_drive/extractor.class:122:in `yield_files\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/content_sources/google_drive/extractor.class:109:in `yield_actions_and_files\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/content_sources/google_drive/extractor.class:97:in `yield_domain_actions_and_files\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/content_sources/google_drive/extractor.class:21:in `yield_document_changes\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/content_sources/base/extractor.class:151:in `block in document_changes\u0027\n\tfrom org/jruby/RubyGenerator.java:104:in `each\u0027\n\tfrom org/jruby/RubyEnumerator.java:396:in `each\u0027\n\tfrom org/jruby/RubyEnumerator.java:414:in `each\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/content_sources/base/extractor.class:158:in `block in document_changes\u0027\n\tfrom /usr/share/enterprise-search/lib/war/gems/gems/statsd-instrument-2.1.1/lib/statsd/instrument.rb:284:in `block in measure\u0027\n\tfrom /usr/share/enterprise-search/lib/war/gems/gems/statsd-instrument-2.1.1/lib/statsd/instrument.rb:53:in `duration\u0027\n\tfrom /usr/share/enterprise-search/lib/war/gems/gems/statsd-instrument-2.1.1/lib/statsd/instrument.rb:284:in `measure\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/stats.class:4:in `measure\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/content_sources/base/extractor.class:148:in `block in document_changes\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/content_sources/base/extractor.class:109:in `block in with_auth_tokens_and_retry\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/content_sources/google_drive/extractor.class:84:in `convert_rate_limit_errors\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/content_sources/base/extractor.class:108:in `block in with_auth_tokens_and_retry\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/content_sources/base/extractor.class:223:in `convert_transient_server_errors\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/content_sources/base/extractor.class:107:in `with_auth_tokens_and_retry\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/content_sources/base/extractor.class:147:in `block in document_changes\u0027\n\tfrom /usr/share/enterprise-search/lib/war/gems/gems/statsd-instrument-2.1.1/lib/statsd/instrument.rb:284:in `block in measure\u0027\n\tfrom /usr/share/enterprise-search/lib/war/gems/gems/statsd-instrument-2.1.1/lib/statsd/instrument.rb:53:in `duration\u0027\n\tfrom /usr/share/enterprise-search/lib/war/gems/gems/statsd-instrument-2.1.1/lib/statsd/instrument.rb:284:in `measure\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/stats.class:4:in `measure\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/content_sources/base/extractor.class:146:in `document_changes\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/work/index.class:13:in `block in run\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/work/abstract_extractor_work.class:83:in `run_with_suspension\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/work/index.class:12:in `run\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/work/abstract_extractor_work.class:121:in `execute\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/workers/extract_worker.class:7:in `block in run\u0027\n\tfrom /usr/share/enterprise-search/lib/war/gems/gems/concurrent-ruby-1.1.8/lib/concurrent-ruby/concurrent/executor/safe_task_executor.rb:24:in `block in execute\u0027\n\tfrom com/concurrent_ruby/ext/SynchronizationLibrary.java:232:in `synchronize\u0027\n\tfrom /usr/share/enterprise-search/lib/war/gems/gems/concurrent-ruby-1.1.8/lib/concurrent-ruby/concurrent/executor/safe_task_executor.rb:19:in `execute\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/config/initializers/concurrent.class:16:in `block in realize\u0027\n\tfrom /usr/share/enterprise-search/lib/war/gems/gems/concurrent-ruby-1.1.8/lib/concurrent-ruby/concurrent/executor/java_executor_service.rb:79:in `run\u0027\n",
            "id": "60d02e4884c212d93133d463",
            "message": "undefined method `[]\u0027 for nil:NilClass",
            "class": "NoMethodError"
          },
          "errorReason": "unknown",
          "completedAt": "2021-06-21T06:14:32Z",
          "durationSeconds": 4.0,
          "documentErrorCount": 0,
          "documentErrors": []
        },
... snip ...
2

Hi @its-ogawa ,

Looking at the fatalException.stack_trace, it looks like Workplace Search can no longer find the authorization information needed to communicate with Google Drive. Can you check a few things?

  • make sure that your Connector Configuration (OAuth Client ID and Client Secret) is still set and correct.
  • make sure that if you are using multiple Enterprise Search servers that they all have the same secret_management.encryption_keys value in your config/enterprise-search.yml
  • make sure that you haven't changed the value of secret_management.encryption_keys since configuring your Connector Configuration for Google Drive.
3

Thanks for the answer.

  • make sure that your Connector Configuration (OAuth Client ID and Client Secret) is still set and correct.

As far as I can see in GoogleCloudPlatform, I can confirm that the Client ID and Client Secret are set.
Can I check them from the WorkplaceSearch administrator screen? I believe I have set those information for sure.

  • make sure that if you are using multiple Enterprise Search servers that they all have the same secret_management.encryption_keys value in your config /enterprise-search.yml

You are using one EnterpriseSearch server.

  • make sure that you haven't changed the value of secret_management.encryption_keys since configuring your Connector Configuration for Google Drive .

The value of secret_management.encryption_keys has not been changed.
It's not a very good configuration, but since you have only one value of secret_management.encryption_keys set, I don't think it will switch.

4

Yes, you should validate that the Client ID and Client Secret that you see in the Google Cloud Platform match what you have configured for your connector. You can get to that view by clicking Settings,

Screen Shot 2021-06-22 at 10.22.25 AM
Screen Shot 2021-06-22 at 10.22.25 AM358×593 31.7 KB

Then click Content source connectors:

Screen Shot 2021-06-22 at 10.22.52 AM
Screen Shot 2021-06-22 at 10.22.52 AM427×535 25.1 KB

Then click the link by Google Drive:

Screen Shot 2021-06-22 at 10.23.22 AM
Screen Shot 2021-06-22 at 10.23.22 AM1327×633 69.9 KB

Then validate that these fields match what you have in the Google Cloud Platform.

Screen Shot 2021-06-22 at 10.23.55 AM
Screen Shot 2021-06-22 at 10.23.55 AM1223×634 77.7 KB

It's not uncommon for customers to make a mistake when manually updating their version of Enterprise Search to not carefully copy their value of secret_management.encryption_keys from the old config file to the new config file. This means that any secrets added in the old version are now un-encryptable in the new version.

1 Like
5

Thank you for the clear explanation.
(I'd like to put this explanation in the official reference!)

When I followed the steps and checked the Client ID and Client Secret, they were blank for some reason.

After checking them in GoogleCloudPlatform and running Save configuration again, another GoogleDrive content was created and the motivation started.

After a while, the synchronization stops and the same message as before appears.

An unknown problem occurred during synchronization. [Check the diagnostic bundle for more information - Message ID: 60d2991884c2129788374894].

The content will be similar, but we will also describe the content of the newly obtained json file.

{
  "general": {
    "version": "7.13.2",
    "revision": "af74d68aafaab071eafa00c2ab628ecf5a1daa27",
    "elasticsearchVersion": "7.11.0"
  },
  "source": {
    "id": "60d2938884c21260d5349327",
    "serviceType": "google_drive",
    "name": "Google Drive (1)",
    "context": "organization",
    "dlpEnabled": false,
    "recentJobs": {
      "incremental": [
        {
          "createdAt": "2021-06-23T01:51:05Z",
          "status": "complete",
          "fatalException": null,
          "errorReason": null,
          "completedAt": "2021-06-23T01:51:20Z",
          "durationSeconds": 15.0,
          "documentErrorCount": 0,
          "documentErrors": []
        }
      ],
      "full": [
        {
          "createdAt": "2021-06-23T01:51:05Z",
          "status": "error",
          "fatalException": {
            "friendly_message": "Unknown problem occurred when syncing.",
            "stack_trace": "/usr/share/enterprise-search/lib/war/gems/gems/google-apis-core-0.2.1/lib/google/apis/core/http_command.rb:235:in `check_status\u0027: Unknown error\n\tfrom /usr/share/enterprise-search/lib/war/gems/gems/google-apis-core-0.2.1/lib/google/apis/core/api_command.rb:136:in `check_status\u0027\n\tfrom /usr/share/enterprise-search/lib/war/gems/gems/google-apis-core-0.2.1/lib/google/apis/core/http_command.rb:194:in `process_response\u0027\n\tfrom /usr/share/enterprise-search/lib/war/gems/gems/google-apis-core-0.2.1/lib/google/apis/core/http_command.rb:310:in `execute_once\u0027\n\tfrom /usr/share/enterprise-search/lib/war/gems/gems/google-apis-core-0.2.1/lib/google/apis/core/http_command.rb:113:in `block in execute\u0027\n\tfrom /usr/share/enterprise-search/lib/war/gems/gems/retriable-3.1.2/lib/retriable.rb:61:in `block in retriable\u0027\n\tfrom /usr/share/enterprise-search/lib/war/gems/gems/retriable-3.1.2/lib/retriable.rb:56:in `retriable\u0027\n\tfrom /usr/share/enterprise-search/lib/war/gems/gems/google-apis-core-0.2.1/lib/google/apis/core/http_command.rb:110:in `block in execute\u0027\n\tfrom /usr/share/enterprise-search/lib/war/gems/gems/retriable-3.1.2/lib/retriable.rb:61:in `block in retriable\u0027\n\tfrom /usr/share/enterprise-search/lib/war/gems/gems/retriable-3.1.2/lib/retriable.rb:56:in `retriable\u0027\n\tfrom /usr/share/enterprise-search/lib/war/gems/gems/google-apis-core-0.2.1/lib/google/apis/core/http_command.rb:102:in `execute\u0027\n\tfrom /usr/share/enterprise-search/lib/war/gems/gems/google-apis-core-0.2.1/lib/google/apis/core/base_service.rb:377:in `execute_or_queue_command\u0027\n\tfrom /usr/share/enterprise-search/lib/war/gems/gems/google-api-client-0.53.0/generated/google/apis/drive_v3/service.rb:1215:in `list_files\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/content_sources/google_drive/extractor.class:193:in `get_files\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/content_sources/google_drive/extractor.class:123:in `block in yield_files\u0027\n\tfrom org/jruby/RubyKernel.java:1442:in `loop\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/content_sources/google_drive/extractor.class:122:in `yield_files\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/content_sources/google_drive/extractor.class:109:in `yield_actions_and_files\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/content_sources/google_drive/extractor.class:105:in `yield_user_actions_and_files\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/content_sources/google_drive/extractor.class:22:in `yield_document_changes\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/content_sources/base/extractor.class:151:in `block in document_changes\u0027\n\tfrom org/jruby/RubyGenerator.java:104:in `each\u0027\n\tfrom org/jruby/RubyEnumerator.java:396:in `each\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/content_sources/base/extractor.class:158:in `block in document_changes\u0027\n\tfrom /usr/share/enterprise-search/lib/war/gems/gems/statsd-instrument-2.1.1/lib/statsd/instrument.rb:284:in `block in measure\u0027\n\tfrom /usr/share/enterprise-search/lib/war/gems/gems/statsd-instrument-2.1.1/lib/statsd/instrument.rb:53:in `duration\u0027\n\tfrom /usr/share/enterprise-search/lib/war/gems/gems/statsd-instrument-2.1.1/lib/statsd/instrument.rb:284:in `measure\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/stats.class:4:in `measure\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/content_sources/base/extractor.class:148:in `block in document_changes\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/content_sources/base/extractor.class:109:in `block in with_auth_tokens_and_retry\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/content_sources/google_drive/extractor.class:84:in `convert_rate_limit_errors\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/content_sources/base/extractor.class:108:in `block in with_auth_tokens_and_retry\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/content_sources/base/extractor.class:223:in `convert_transient_server_errors\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/content_sources/base/extractor.class:107:in `with_auth_tokens_and_retry\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/content_sources/base/extractor.class:147:in `block in document_changes\u0027\n\tfrom /usr/share/enterprise-search/lib/war/gems/gems/statsd-instrument-2.1.1/lib/statsd/instrument.rb:284:in `block in measure\u0027\n\tfrom /usr/share/enterprise-search/lib/war/gems/gems/statsd-instrument-2.1.1/lib/statsd/instrument.rb:53:in `duration\u0027\n\tfrom /usr/share/enterprise-search/lib/war/gems/gems/statsd-instrument-2.1.1/lib/statsd/instrument.rb:284:in `measure\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/stats.class:4:in `measure\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/content_sources/base/extractor.class:146:in `document_changes\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/work/index.class:13:in `block in run\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/work/abstract_extractor_work.class:83:in `run_with_suspension\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/work/index.class:12:in `run\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/work/abstract_extractor_work.class:121:in `execute\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/lib/connectors/workers/extract_worker.class:7:in `block in run\u0027\n\tfrom /usr/share/enterprise-search/lib/war/gems/gems/concurrent-ruby-1.1.8/lib/concurrent-ruby/concurrent/executor/safe_task_executor.rb:24:in `block in execute\u0027\n\tfrom com/concurrent_ruby/ext/SynchronizationLibrary.java:232:in `synchronize\u0027\n\tfrom /usr/share/enterprise-search/lib/war/gems/gems/concurrent-ruby-1.1.8/lib/concurrent-ruby/concurrent/executor/safe_task_executor.rb:19:in `execute\u0027\n\tfrom /usr/share/enterprise-search/lib/war/connectors/config/initializers/concurrent.class:16:in `block in realize\u0027\n\tfrom /usr/share/enterprise-search/lib/war/gems/gems/concurrent-ruby-1.1.8/lib/concurrent-ruby/concurrent/executor/java_executor_service.rb:79:in `run\u0027\n",
            "id": "60d2991884c2129788374894",
            "message": "Unknown error",
            "class": "Google::Apis::TransmissionError"
          },
          "errorReason": "unknown",
          "completedAt": "2021-06-23T02:14:49Z",
          "durationSeconds": 1424.0,
          "documentErrorCount": 0,
          "documentErrors": []
        }
      ]
    },
    "oauthTokenAgeSeconds": 2384,
    "oauthScopes": [
      "email",
      "profile",
      "https://www.googleapis.com/auth/drive.readonly",
      "https://www.googleapis.com/auth/drive.metadata.readonly",
      "https://www.googleapis.com/auth/admin.directory.group.readonly",
      "https://www.googleapis.com/auth/admin.directory.user.readonly"
    ]
  }
}

Is there a solution for these?

6

Has it continued to fail with this error? This tells us that Google had a problem, and is outside of the control of our code. Likely, Google's API had a brief outage of some sort.

7

I see.

It's a problem with the Google API, isn't it?
Indeed, when I checked again a few days later, the number of items in the content had increased, so in the case of this error, there is nothing wrong with WorkplaceSearch, so it might be a good idea to give it some time and see what happens.

8

Sorry again and again, but I am getting a different error.
How should I interpret this?

{
  "general": {
    "version": "7.13.2",
    "revision": "af74d68aafaab071eafa00c2ab628ecf5a1daa27",
    "elasticsearchVersion": "7.11.0"
  },
  "source": {
    "id": "60d2938884c21260d5349327",
    "serviceType": "google_drive",
    "name": "Google Drive (1)",
    "context": "organization",
    "dlpEnabled": false,
    "recentJobs": {
      "incremental": [
        {
          "createdAt": "2021-06-24T23:53:54Z",
          "status": "complete",
          "fatalException": null,
          "errorReason": null,
          "completedAt": "2021-06-24T23:53:58Z",
          "durationSeconds": 4.0,
          "documentErrorCount": 0,
          "documentErrors": []
        },
        ... snip ...
      ],
      "delete": [
        {
          "createdAt": "2021-06-24T23:05:52Z",
          "status": "error",
          "fatalException": {
            "friendly_message": "Unknown problem occurred when syncing.",
            "stack_trace": "0: Read timed out\n",
            "id": "60d52cd784c212d931aaeb22",
            "message": "Read timed out",
            "class": "Faraday::TimeoutError"
          },
          "errorReason": "unknown",
          "completedAt": "2021-06-25T01:09:43Z",
          "durationSeconds": 7431.0,
          "documentErrorCount": 0,
          "documentErrors": []
        },
        ... snip ...
      ],
      "full": [
        {
          "createdAt": "2021-06-24T19:29:44Z",
          "status": "error",
          "fatalException": {
            "friendly_message": "Failed to save data to search index.",
            "stack_trace": ... snip ...,
            "message": "Connectors::Work::AbstractExtractorWork::PublishingFailedError",
            "class": "Connectors::Work::AbstractExtractorWork::PublishingFailedError"
          },
          "errorReason": "indexing_failed",
          "completedAt": "2021-06-24T23:16:00Z",
          "durationSeconds": 13576.0,
          "documentErrorCount": 1001,
          "documentErrors": [
            {
              "error_id": "60d5122284c212342b9ecaf7",
              "error_class": "Connectors::Work::AbstractExtractorWork::PublishingFailedError",
              "error_message": "Exception: Connectors::Work::AbstractExtractorWork::PublishingFailedError: Connectors::Work::AbstractExtractorWork::PublishingFailedError: Context: {:message_id\u003d\u003e\"60d5122284c212342b9ecaf7\"}",
              "stack_trace": ... snip ...
            },
            ... snip ...
          ]
        },
        ... snip ...
      ]
    },
    "oauthTokenAgeSeconds": 153,
    "oauthScopes": [
      "email",
      "profile",
      "https://www.googleapis.com/auth/drive.readonly",
      "https://www.googleapis.com/auth/drive.metadata.readonly",
      "https://www.googleapis.com/auth/admin.directory.group.readonly",
      "https://www.googleapis.com/auth/admin.directory.user.readonly"
    ]
  }
}
9

Can you re-send with the stacktraces included, instead of ...snip....? That will help me to identify the root cause. Don't worry about it being unreadable, the discuss formatting will put all these on a single line.

From what you've shared, all I can be sure of is that something went wrong while attempting to index data into elasticsearch. It could be an Elasticsearch health issue, or it could be a bug in Workplace Search, or it could be transient. Does the error continue to happen? It looks like the most recent Incremental job completed successfully.

10

My apologies.
There was a lot of text, so I took the liberty of cutting it out.
The stack_trace is as follows.

11

After watching for a while, the error was no longer displayed and the status of the sync changed to Complete.
I was relieved.

@Sean_Story

Is it correct to assume that WorkplaceSearch will run the sync repeatedly until the status becomes complete?

Also, what should I look for to distinguish between an error that time will resolve and an error that needs some action, like this one?

I'd like to close this topic soon. Thank you in advance.

12

Hi @its-ogawa ,

Sorry that I missed responding to your full post of the connector diagnostics. There's unfortunately not a smoking gun in there that I can see. I would have recommended that you take the contextual message IDs you can find in there (like Context: {:message_id\u003d\u003e\"60d4dc4b84c2129788871790\"}") and grep for those ids in your logs, looking for other stack traces or errors immediately before those. Something was causing your asynchronous publishing tasks to fail semi-regularly, which tripped the Max Error Ratio circuit breaker: Common source synchronization issues | Workplace Search documentation [8.11] | Elastic. If you see this again, you could try increasing either of:

workplace_search.content_source.max_error_ratio #defaults to 0.15
# OR
workplace_search.content_source.error_ratio_window_size #defaults to 100

Sort of. Workplace search will continually sync with your standard content sources. Full re-syncs every 72 hours, "latest changes" every 2 hours, "deleted documents" every 6 hours, and "permissions updates" (if document level permissions are enabled for your source) every 5 minutes. If one of those jobs fails, another job of the same type will restart sooner than it would have otherwise. So yes, sometimes jobs fail for transient reasons (network issues, service unavailaility, etc) and will succeed on a consecutive retry. Sometimes the issue is not transient, and you will have the same failure again and again until the underlying issue is resolved. I suggest you consult Common source synchronization issues | Workplace Search documentation [8.11] | Elastic with each specific failure to get a feeling for what the issue may be. If it's not in that list or you are still having trouble resolving an issue, you may want to consider talking to us about a subscription in order to access full-time support and troubleshooting.

13

@Sean_Story ,

Thank you for your answer.

I am sorry for rushing to answer.
I am certainly not in a position to take on full time support.
I would appreciate it if you could answer when you feel like it.
(I hope you will except the forum from time to time.......)

What are the logs you are referring to here?
There are some logs in /var/log/enterprise-search, but they do not contain document IDs (e.g. 60d4dc4b84c2129788871790).

In particular, the document ID does not seem to be available through the API for the basic license, so we do not know what document the document ID refers to.
Is this error something that basic license users can't handle? If so, I am very disappointed.

14

Hi @its-ogawa,

Are you using docker? If so, then yes, the logs would be in /var/log/enterprise-search. However, the logs "roll over", so since it has been quite a while since you first posted about this error, those log files have likely removed themselves. Something to just know for next time, since it sounds like your syncs are working now?

You are correct that the basic license does not include the Search API, which is how you would use a document ID to look up a document. However the message_id we were discussing above is not a "document ID" but a "log message ID". That ID is used to help you tie your error message in the UI to the stack trace in your connector diagnostics JSON, to the relevant lines in your logfile.

15

I installed the package on my own CentOS instead of Docker.
(I guess the configuration isn't too different from Docker)

Your guess is quite correct. Too much time has passed since the first error.

What I need to know is how to read the json file that WorkplaceSearch issues on error.
This is where the log message ID (which I mistook for the document ID in my last post) shows up in which log file, and where the error details are listed.

In my /var/log/enterprise-search, I have 8 different logs being output.

  • app-server.log
  • connectors.log
  • filebeat
  • gc.log
  • init.log
  • stats.log
  • system.log
  • worker.log

Can I look for the log message ID in the json file in app-server.log (?) and assume that the error details are in there? and assume that the error details are listed there?

16

Hi @its-ogawa ,

For errors you see in the content source UI and the connectors diagnostics json, you're most likely to see the corresponding message IDs in connectors.log or worker.log. But you can use grep to search over all the log files just to be sure. app-server.log will typically have the logs relevant to client network traffic to Enterprise Search, which can be helpful with troubleshooting any errors you get when interacting with our APIs, or if web pages are not loading correctly.

17

@Sean_Story
I see!

Thanks for the good quality answers.

If there is any problem, I will first and foremost refer to connectors.log, worker.log and app-server.log to pursue the cause.
And if I can't figure it out, I would like to consult the Elastic forum (by starting a new topic).

So far, there seems to be no errors or warnings in these logs.
I am no longer able to reproduce the situation when I started the topic. My apologies.
I would like to close this topic.

Thank you for your help in getting this resolved.

1 Like
18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.