ElasticSearch partial re-index with Logstash

Elastic Stack Logstash
1

Hi All!

I'm trying to re-index data from one ES cluster to another with { "match_all": {} } query, but I'd like to limit the result set by a certain number. Currently, all "total" hits are being fetched from the source ES cluster.
Is there a way to stop the input when reaching a certain number of documents?
The "size" parameter of the ES input plugin is applicable for the scrolling only.

Thank you!

2

There is not.

3

Thank you for the fast reply!
Do you think it'd make sense to implement a new config parameter (e.g. "count") to achieve that? Thanks!

4

Maybe, but what is the use case, why do you only need a certain number of things?

5

For what it is worth _reindex in Elasticsearch 2.3+ supports limiting the number of documents copied. I added that because it makes it very convenient to play with smaller data sets.

6

@warkolm, the use case - to split one big index into chunks, in order to analyse them separately.
@nik9000, thank you for the hint! You are the genius! :slight_smile:

7

Can you possibly give a small example of implementing _reindex? It sounds like it's what I need, but I can't find any documentation on the website.

8

It's an ES thing - https://www.elastic.co/guide/en/elasticsearch/reference/2.3/docs-reindex.html

9

Great, thanks for the quick reply, and link!