How to use elasticsearch input plugin in logstash to get more than 10000 results from ES

Elastic Stack Logstash
1

Hello,

I want to copy the data in a distant Es (which I have a limited access ) to another ES where I have full rights.
So,the input Es index contains 50000 documents , and I manage to get only 10000 hits using the following configuration:

input {

elasticsearch {
hosts => ["xxxx:9200"]
index => "forxx"
user => "kibxx"
password =>"xxx!"
query => '{ "query": {"match_all": {}} }'
tags => "table_elastic_to_elastic"
ssl_certificate_verification => false
ssl => true
size => 10000

}
output {

stdout {}
if "table_elastic_to_elastic" in [tags] {
elasticsearch {
    index => "newindex"
    document_id => "%{ID}"
    hosts => ["localhost:9200"]
    }
}

}**

2 Likes
2

Is it because you're defining the size to 10000?

3

If don't specify size=10000 , logstash will use the default value which is equal to 1000

4

increase the size to 50000

5

the maximum size is 10000

6

use the scroll API, search the forums here and you will get answers. Its always a good practice to search the forum before posting or google it :slight_smile:
If still need help please ask here

7

if I don't specify scroll , then logstash will use the default value which is equal to "1m".
I've searched google and forums but I didn't find any answers

1 Like
8

Have you tried on Postman perform a GET to see if you're receiving 50k or just 10k?

9

Any reason why you can't use Reindex from Remote instead of Logstash?

10

I chose Logstash to run the job everyday at a certain time using Logstash "schedule"

11

You could use cron or similar to schedule a script that calls the reindex API once a day. The reindex API supports queries, which would allow you to only transfer those entries that were done since the last run (if your data contains timestamps).

When transferring large result sets, you should also look into using the scroll API.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.