Engine Name is already Taken (second occurrence)

Elastic Search
1

I previously posted about this in June Engine Name is already Taken - Elastic Enterprise Search - Discuss the Elastic Stack
where @Artem_Shelkovnikov was helpful in resolving the issue

Everything has been working great until it re-occurred this morning, ruining a sales presentation of our product
I've used the same technique in the original post to resolve the issue, however we don't want this to happen again.

  1. Can this issue be permanently resolved by the Elastic team; or
  2. We delete and re-create our engines every day because there is no bulk delete of documents; can such an API endpoint be created?
2

Hi @Rav_Panchalingam,

I'm sorry about the presentation. That's awful to hear.

We do have an API to delete documents: Documents API | App Search documentation [8.11] | Elastic.

This takes a JSON array of strings as document IDs. Does that work for you? How many documents are you trying to delete? Everything in the engine? I'm not sure about the limits on the number of documents you can delete with that API actually, but as a warning, if the documents are large or you're trying to delete a very large number of documents, that endpoint may not be very performant.

I'm surprised with the original error you're receiving, though. Are you able to reproduce this error in a consistent way? Are you checking the responses of the delete API calls? Curious to know more about this. I wouldn't expect the records to exist in the index still. Artem mentioned:

It seems more likely that isn't something wrong with the deletion side of things, but I'm interested to dig further.

Best
Brian

3

Hi @Brian_McGue sorry for late response, we ended up doing a bunch of undesirable workarounds to prevent this from happening again, but it would be very useful to have a bulk delete or clear index endpoint. Currently the delete documents endpoint for App Search has a 100 document limit. We have multiple engines with between 1k and 200k records. The 1k record engines are no issue but the large ones can take hours to delete in this way (seems that the larger the engine the longer it takes to add or delete). Delete requests on the large engines can take up to 30s per 100 documents.

4

similarly, a bulk index endpoint would be very useful, with ability to ingest a compressed file attachment or something like that. 100 document limit is a pain point for us.

5

Hi @Rav_Panchalingam,

It's worth noting that as of 8.2, App Search has the ability to work with Elasticsearch indices directly: Elasticsearch index engines (beta) | Elastic App Search Documentation [8.4] | Elastic. You can create an Elasticsearch index-based engine: Create Elasticsearch index engines | Elastic App Search Documentation [8.4] | Elastic.

Note, there is only limited support for Elasticsearch index-based engines currently, all of which should be detailed in the links provided above, but this may help you with your issues with bulk indexing and deleting documents because you would be able to interact with the Elasticsearch indices directly.

If that is too big of a change, there may be a more simple workaround here that is not officially supported to delete the documents in the underlying Elasticsearch index that is used for an App Search engine. Please test this out in a dev environment or on dev engines/indices first. Things like Curations will still exist that would otherwise be deleted.

If this is something you're willing to try, you can find the underlying Elasticsearch index by using the engine name: .ent-search-engine-documents-{engine_name}. From there, you can use the Delete by query API | Elasticsearch Guide [8.4] | Elastic with a Match all query | Elasticsearch Guide [8.4] | Elastic to remove all of the documents within the index. Since this is using Elasticsearch directly, it should be very performant.

Hope this helps,
Brian

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.