[App Search] - Ingesting Wrong Documents Issue

[aisyaharifin (Siti Nuraisyah)]

Hello, I have a case where the data that I've been ingested got detected as wrong data.

The data have been classified into MATERIAL and SERVICE type of data.

The wrong data resides under SERVICE classification with a total of 274,552 documents to be exact and it is detected that the data value has been wrongly extracted by the team.

My plan was to either update the data or delete the data with SERVICE classification and ingest a new one.

I'm not sure if I can update it according to the new json file.

But I see that deleting it might be the option or maybe not.

My case would be just like this post :

How to delete documents by a certain field?

Hello, I have an App Search engine and I need to delete documents in it by a certain field value. I receive JSON files that look something like this: { "entries": [ { "id": "park_rocky-mountain", "category": "national-parks", "title": "Rocky Mountain", "nps_link": "https://www.nps.gov/romo/index.htm", "states": [ "Colorado" ], "visitors": 4517585, "world_heritage_site": false, "location": "40.4,-105.58", "acres": 265795.…

When I reached out to the support the idea was to filter the data by query it through item_type = SERVICE, get the IDs and delete the documents referencing the IDs.

The above post also seems to be said the same thing :

It seems like the option is still needed to reference through the document IDs which I think it's impossible to list out 200k+ IDs here.

It's not really a good solution.

Is there any other options for me?

Or the best way is to wipe out the whole documents and ingest a new data?

If so, do I need to create a new search engine to ingest the data back or I can just simply delete all the documents in my current search engine without referencing to the IDs?

Thank you.

[Carlos_D (Carlos)]

Hi @aisyaharifin :

App Search does not provide an API for deleting documents that does not depend on the ID. You will need document IDs in order to delete documents.

I think you can follow one of these two options:

• Delete the engine and re-ingest all data
• Delete the SERVICE data in the engine and re-ingest it

For option 2, you will need a script or small program that:

• Performs search using a value filter on the SERVICE field.
• For each ID retrieved on the search, do a document delete for that ID.

The best option will depend on how long does it take to re-ingest your data vs the time you will need to create the delete script + re-ingest the data

Hope that helps!