[App Search] - Ingesting Wrong Documents Issue

Hello, I have a case where the data that I've been ingested got detected as wrong data.

The data have been classified into MATERIAL and SERVICE type of data.

The wrong data resides under SERVICE classification with a total of 274,552 documents to be exact and it is detected that the data value has been wrongly extracted by the team.

My plan was to either update the data or delete the data with SERVICE classification and ingest a new one.

I'm not sure if I can update it according to the new json file.

But I see that deleting it might be the option or maybe not.

My case would be just like this post :

When I reached out to the support the idea was to filter the data by query it through item_type = SERVICE, get the IDs and delete the documents referencing the IDs.

The above post also seems to be said the same thing :

It seems like the option is still needed to reference through the document IDs which I think it's impossible to list out 200k+ IDs here.

It's not really a good solution.

Is there any other options for me?

Or the best way is to wipe out the whole documents and ingest a new data?

If so, do I need to create a new search engine to ingest the data back or I can just simply delete all the documents in my current search engine without referencing to the IDs?

Thank you.

Hi @aisyaharifin :

App Search does not provide an API for deleting documents that does not depend on the ID. You will need document IDs in order to delete documents.

I think you can follow one of these two options:

  • Delete the engine and re-ingest all data
  • Delete the SERVICE data in the engine and re-ingest it

For option 2, you will need a script or small program that:

The best option will depend on how long does it take to re-ingest your data vs the time you will need to create the delete script + re-ingest the data

Hope that helps!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.