AppSearch Swiftype::Documents::DuplicateKeyError Self-Managed

Elastic Search
1

Hi we're trying to install a self managed version of App Search on our server, however when trying to connect to the API to index documents it's returning me the following error:
["Internal server error. Please email support@swiftype.com if this error persists."]

In the server logs, it looks like there is some sort of duplicate key error, but I can't find anything about this online.

It does let 1 or 2 API requests through before throwing this error. You can see the server logs here: http://134.209.185.32/appsearcherror.txt

Anyone have any ideas on how we could fix this?

Thanks, Harry.

2

Hello Harry,

Sorry you are experiencing issues with the product.

Could you please check which App Search version you're running? The easiest way to get that information from a running instance would be to hit the health API and share the results: http://localhost:3002/api/as/v1/internal/health (or whatever the URL is for your installation).

Thank you!

P.S. Thank you for the detailed error message, we are looking to improve error messaging in cases like yours.

3

Looked closer at the issue and managed to reproduce it. I see a number of documents in that documents list that have duplicate ID field values. That will not work since all documents are expected to have a unique id (if provided).

We're going to look into improving error messages for cases like that, but for now I'd recommend you ensure uniqueness of those document ids to avoid the error.

Thank you for your report!

4

Just confirmed, that in App Search 7.5.0 (should be out within the next couple of weeks) the behavior should be fixed and duplicate errors in batch indexing requests will be handled as expected.

5

Hi @oleksiy-elastic

Thanks, checking for duplicates has fixed this issue.

Cheers,

  • Harry.
6

Hi Oleksiy,

I've been encountering this problem again, inconsistently. I have been checking for duplicates before sending my documents to be indexed, and have confirmed that there are no duplicates being sent, however I am still getting the internal server error.

I have checked the logs and it seems to be spitting the duplicate key error, here is what I could find:

Swiftype::Documents::DuplicateKeyError (Swiftype::Documents::DuplicateKeyError):


/tmp/jetty-127.0.0.1-3002-app-search.war-_-any-13963166313493986293.dir/webapp/WEB-INF/lib/swiftype/document_storage/backends/elasticsearch_backend.class:369:in `block in save_constraints!'


org/jruby/RubyHash.java:1419:in `each'


--


[2019-12-02T14:22:22.329+00:00][3866287][14670][app-server][ERROR]: [004b62a6-89d3-4694-9319-188fb31fa070] Swiftype::ExceptionTracking.capture_type: exception, {},





Swiftype::Documents::DuplicateKeyError (Swiftype::Documents::DuplicateKeyError):


/tmp/jetty-127.0.0.1-3002-app-search.war-_-any-13963166313493986293.dir/webapp/WEB-INF/lib/swiftype/document_storage/backends/elasticsearch_backend.class:369:in `block in save_constraints!'


org/jruby/RubyHash.java:1419:in `each'


And:

[2019-12-02T14:22:54.204+00:00][3866291][2250][worker][INFO]: [ActiveJob] [Work::Engine::IndexAdder] [92b250dea9c303f036b84f9236eaa1d1e2950325] Ignoring duplicate job class=Work::Engine::RefreshDocumentCounts, id=1b066b8563b4a99f958b1456e98bebc3fef934e2, args=["5de5153e6ab19b4bdd01c029"]

Do you have any other ideas as to why this might be happening? I am using the documents api to sent the documents, and in batches of around 70 documents.

Thanks in advance for any help!

  • Harry.
7

It seems this same error:
["Internal server error. Please email support@swiftype.com if this error persists."]
And the DuplicateKey server error are thrown for all document id issues.

If anyone else encounters the Internal server error, ensure your document id's match all of the rules /errors outlined here: https://swiftype.com/documentation/app-search/api/documents#errors

My issue was some id's contained either capital letters or '-'s and Swiftype throws the same error for all these issues.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.