[NEST] How to move documents/aggregations to another index?

Elastic Stack Elasticsearch
1

I found that it 's possible, but I can't find the solution in the C# library NEST...

I want to aggregate documents by a certain hash value, and move them to a different index, how would I do that in NEST?

2

I did some more research, I think it's better to just give the documents an "archived" field instead of moving the documents to a different index...

3

I'm not sure if you still need it, but I guess what you need to use in order to move data that has a specific value from an index to another is to use the reindex API:

https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html

You can specify in the reindex API a query and all the documents matching this query will be moved into a new index.

4

I did already come across that article before I posted this... I was wondering if NEST did support this as well in their library...

But moving documents from one index to another requires reindexing of the indices right? That would take a toll on the performance maybe?

5

NEST supports the Reindex API through the .ReindexOnServer() method (and the async variant, ReindexOnServerAsync()).

ReIndex API won't move documents, but reindex them from one index to another. After reindexing, they will still exist in the source index. You could look to delete them using Delete By Query API which NEST supports through .DeleteByQuery(), although be aware that deleting a lot of documents from an index can be a relatively expensive operation compared to reindexing documents from one source index into two new indices.

6

Thanks for the extensive answer. I don't think reindexing, moving, deleting documents is a good idea in my situation, giving the documents an extra Boolean parameter is probably a better idea...

The idea is to "archive" certain documents so that I can apply a filter on them to view only non-archived, archived or both... Throwing the documents around will probably make my life a lot harder, and will probably take a lot of resources since the dataset is going to be large...

I thought moving the documents to a second index would make the performance of a query on the main most important/most used index better since I keep the data set of this index small, but I don't think a large dataset is a big issue for Elasticsearch...

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.