Logstash seems to copy the index indefinitely

[pain368 (Mr)]

Hello ELK team,

This is my first thread so i hope will be good enough to solve my problem.

I have One primary cluster(A), and today i created second cluster(B) only for Machine Learning purpose ( i don;t want to connect them each other). Also i don't use license version on primary cluster, but i start 30-days trial license on B cluster

Now is my problem:
I use logstash to copy just one index from cluster-A, which has 42,5Gb and 41240087 Docs count (Info from "Index Managent" tab in Kibana):
here is the config of Logstsash

input {
    elasticsearch {
        hosts => ["x.x.x.1", "x.x.x.2", "x.x.x.3"]
        user => "userX",
        password => "passX",
        index => "index-from-cluster-A",
        dockinfo => true
    }
}
output {
    elasticsearch {
        hosts => ["x.x.x.5", "x.x.x.6", "x.x.x.7", "x.x.x.8"]
        index => "copy-index-from-cluster-A-%{+YYYY.MM.dd}"
    }
}

The problem is that when the index reaches a similar size on cluster B as on cluster A, the logstash does not stop its work at all. Additionally, when the number of documents is the same on cluster B as on cluster A, the logstash still does not stop copying.

When the index on the B cluster reaches 100Gb, I am forced to turn off the logstash, because it does not make sense ... it seems to me that the index should be exactly the same (I expect this), I will add that on cluster-A the indexes are created from a template

what am I doing wrong ?

At this moment i don't want to use CCR, i used reindexing API but it won't work properly because there is some error in docs inside indices

My version on evry elastic nodes, Kibana, and Logstash are in 7.11.2

[pain368 (Mr)]

There is an idea ??

[stephenb (Stephen Brown)]

Hi @pain368 welcome to the community.

Could you try putting a single host and see what happens?

hosts => "x.x.x.1"

Also with a large data set you should probably use the scroll like in the example in the docs

input {
      elasticsearch {
        hosts => "es.production.mysite.org"
        index => "mydata-2018.09.*"
        query => '{ "query": { "query_string": { "query": "*" } } }'
        size => 500
        scroll => "5m"
        docinfo => true
        docinfo_target => "[@metadata][doc]"
      }
    }
    output {
      elasticsearch {
        index => "copy-of-production.%{[@metadata][doc][_index]}"
        document_type => "%{[@metadata][doc][_type]}"
        document_id => "%{[@metadata][doc][_id]}"
      }
    }

[pain368 (Mr)]

Hello @stephenb thx for replay :), yes it is the right config

input {
    elasticsearch {
        hosts => ["x.x.x.1"]
        user => "userX",
        password => "passX",
        index => "index-from-cluster-A",
        dockinfo => true
    }
}
output {
    elasticsearch {
        hosts => ["x.x.x.5",]
        index => "copy-index-from-cluster-A-%{+YYYY.MM.dd}"
        document_type => "%{[@metadata][doc][_type]}"
        document_id => "%{[@metadata][doc][_id]}"
    }
}