# Logstash seems to copy the index indefinitely

**URL:** <https://discuss.elastic.co/t/logstash-seems-to-copy-the-index-indefinitely/297698>\
**Category:** Logstash\
**Created:** [February 20, 2022, 2:43pm UTC](https://discuss.elastic.co/t/logstash-seems-to-copy-the-index-indefinitely/297698 "2022-02-20T14:43:32Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![pain368](https://avatars.discourse-cdn.com/v4/letter/p/9d8465/32.png) [@pain368](https://discuss.elastic.co/u/pain368)\
**Post date:** [February 20, 2022, 2:43pm UTC](https://discuss.elastic.co/t/logstash-seems-to-copy-the-index-indefinitely/297698/1 "2022-02-20T14:43:33Z")

</div>

Hello ELK team,

This is my first thread so i hope will be good enough to solve my problem.

I have One primary cluster(A), and today i created second cluster(B) only for Machine Learning purpose ( i don;t want to connect them each other). Also i don't use license version on primary cluster, but i start 30-days trial license on B cluster

Now is my problem:  
I use logstash to copy just one index from cluster-A, which has 42,5Gb and 41240087 Docs count (Info from "Index Managent" tab in Kibana):  
here is the config of Logstsash

```auto
input {
    elasticsearch {
        hosts => ["x.x.x.1", "x.x.x.2", "x.x.x.3"]
        user => "userX",
        password => "passX",
        index => "index-from-cluster-A",
        dockinfo => true
    }
}
output {
    elasticsearch {
        hosts => ["x.x.x.5", "x.x.x.6", "x.x.x.7", "x.x.x.8"]
        index => "copy-index-from-cluster-A-%{+YYYY.MM.dd}"
    }
}

```

The problem is that when the index reaches a similar size on cluster B as on cluster A, the logstash does not stop its work at all. Additionally, when the number of documents is the same on cluster B as on cluster A, the logstash still does not stop copying.

When the index on the B cluster reaches 100Gb, I am forced to turn off the logstash, because it does not make sense ... it seems to me that the index should be exactly the same (I expect this), I will add that on cluster-A the indexes are created from a template

what am I doing wrong ?

At this moment i don't want to use CCR, i used reindexing API but it won't work properly because there is some error in docs inside indices

My version on evry elastic nodes, Kibana, and Logstash are in 7.11.2

---

<div class="post-metadata">

**Author:** ![pain368](https://avatars.discourse-cdn.com/v4/letter/p/9d8465/32.png) [@pain368](https://discuss.elastic.co/u/pain368)\
**Post date:** [February 21, 2022, 10:49am UTC](https://discuss.elastic.co/t/logstash-seems-to-copy-the-index-indefinitely/297698/2 "2022-02-21T10:49:05Z")

</div>

There is an idea ??🙂

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 21, 2022, 3:10pm UTC](https://discuss.elastic.co/t/logstash-seems-to-copy-the-index-indefinitely/297698/3 "2022-02-21T15:10:42Z")

</div>

Hi @pain368 welcome to the community.

Could you try putting a single host and see what happens?

`hosts => "x.x.x.1"`

Also with a large data set you should probably use the scroll like in the example in the [docs](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-elasticsearch.html#plugins-inputs-elasticsearch-docinfo)

```auto
input {
      elasticsearch {
        hosts => "es.production.mysite.org"
        index => "mydata-2018.09.*"
        query => '{ "query": { "query_string": { "query": "*" } } }'
        size => 500
        scroll => "5m"
        docinfo => true
        docinfo_target => "[@metadata][doc]"
      }
    }
    output {
      elasticsearch {
        index => "copy-of-production.%{[@metadata][doc][_index]}"
        document_type => "%{[@metadata][doc][_type]}"
        document_id => "%{[@metadata][doc][_id]}"
      }
    }

```

---

<div class="post-metadata">

**Author:** ![pain368](https://avatars.discourse-cdn.com/v4/letter/p/9d8465/32.png) [@pain368](https://discuss.elastic.co/u/pain368)\
**Post date:** [February 22, 2022, 2:23pm UTC](https://discuss.elastic.co/t/logstash-seems-to-copy-the-index-indefinitely/297698/4 "2022-02-22T14:23:00Z")

</div>

Hello @stephenb thx for replay :), yes it is the right config

```auto
input {
    elasticsearch {
        hosts => ["x.x.x.1"]
        user => "userX",
        password => "passX",
        index => "index-from-cluster-A",
        dockinfo => true
    }
}
output {
    elasticsearch {
        hosts => ["x.x.x.5",]
        index => "copy-index-from-cluster-A-%{+YYYY.MM.dd}"
        document_type => "%{[@metadata][doc][_type]}"
        document_id => "%{[@metadata][doc][_id]}"
    }
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 22, 2022, 2:23pm UTC](https://discuss.elastic.co/t/logstash-seems-to-copy-the-index-indefinitely/297698/5 "2022-03-22T14:23:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.