Looking for an alternative to reindex and collapse

[andni233 (Andreas Nilsson)]

Hello!

I have an index that contains a lot of "raw" data, around 40.000 documents.

I want to refine this data a bit, and store the refined data in a different index. For this purpose, I came up with this search query to find the documents that I want to store in the other index:

{
    "query": {"match": {"Selector": "selector"}},
    "collapse": {
        "field": "Identifier",
        "inner_hits": {
            "name": "document_to_reindex",
            "size": 1,
            "sort": [{"FirstTimestmap": "desc"}, {"SeocndTimestamp": "desc"}],
        },
    },
}

This works and gives me the documents I need.

When I want to move the search results to the other index, I figure I would make use of the reindex API. However, this does not work with the above query as it uses collapse. My second though was to fetch the data to a computer somewhere using the above query and then upload it to the new index from there. This does not work due to the search hitting more than 10.000 documents.

Do you have any advice on how I can proceed with this issue? Can collapse be used for this task?