# Data replication from one ES to another using Logstash

**URL:** <https://discuss.elastic.co/t/data-replication-from-one-es-to-another-using-logstash/149690>\
**Category:** Logstash\
**Created:** [September 24, 2018, 2:11pm UTC](https://discuss.elastic.co/t/data-replication-from-one-es-to-another-using-logstash/149690 "2018-09-24T14:11:28Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dolph\_2709](https://avatars.discourse-cdn.com/v4/letter/d/258eb7/32.png) [@Dolph\_2709](https://discuss.elastic.co/u/Dolph_2709)\
**Post date:** [September 24, 2018, 2:11pm UTC](https://discuss.elastic.co/t/data-replication-from-one-es-to-another-using-logstash/149690/1 "2018-09-24T14:11:28Z")

</div>

Hello,

I need to replicate data from old ES (ver. 1.1) instance to latest ELK.  
The older ES is part of IBM Mobile First solution, I can access it through REST API.

This is a Logstash's config script I use on destination server (which has latest version of ELK):

> input {  
> elasticsearch {  
> hosts =\> ["1X.XX.1XX.1XX:9500"]  
> index =\> "worklight"  
> scroll =\> "10m"  
> size =\> 4000  
> query =\> '{ "query": { "range": { "timestamp": {"gt":"now-10m/m", "lte" :"now" } } } }'  
> docinfo =\> true  
> schedule =\> "\* \* \* \* \*"  
> }  
> }
> 
> ```
> filter{ 
> date {
> match => ["timestamp", "UNIX_MS"]
> }
> 
> mutate 
> {
> add_field => { "log_type" => "%{[@metadata][_type]}" }
>       
> }
> mutate
> {
> lowercase => ["log_type"]
> }
> 
> ```
> 
> }
> 
> output {  
> if ( [log\_type] == "customdata" or [log\_type] == "mfpapplogs" ) {
> 
> ```
> elasticsearch {
> hosts=>["1x.XX.XX.1XX:9200"]
> index => "mfp_%{[log_type]}-%{+YYYY.MM.dd}"
> document_type => "%{[@metadata][_type]}"
> document_id => "%{[@metadata][_id]}"
> }
> }
> 
> ```
> 
> }

It worked OK, but after few days when indexes close to 3 mil records, it starts to "slowdown" Kibana on destination server and even stopped ES instance.

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [September 24, 2018, 10:05pm UTC](https://discuss.elastic.co/t/data-replication-from-one-es-to-another-using-logstash/149690/2 "2018-09-24T22:05:38Z")

</div>

Your performance problems appear to be related to your Elasticsearch cluster, and you'll likely be better off asking about it in the Elasticsearch forum.

When you do, please include stats about your Elasticsearch cluster, including node-count, hardware information, stats on your number of indexes, number of shards, documents-per-index, and query patterns.

If Elasticsearch was stopped due to load, log messages from Elasticsearch in that timeframe could also be very helpful in figuring out why.

---

<div class="post-metadata">

**Author:** ![Dolph\_2709](https://avatars.discourse-cdn.com/v4/letter/d/258eb7/32.png) [@Dolph\_2709](https://discuss.elastic.co/u/Dolph_2709)\
**Post date:** [September 25, 2018, 6:38pm UTC](https://discuss.elastic.co/t/data-replication-from-one-es-to-another-using-logstash/149690/3 "2018-09-25T18:38:48Z")

</div>

I think that the problem is in elasticsearch plugin (input)

> input {  
> elasticsearch {  
> hosts =\> ["1X.XX.1XX.1XX:9500"]  
> index =\> "worklight"  
> scroll =\> "10m"  
> size =\> 4000  
> query =\> '{ "query": { "range": { "timestamp": {"gt":"now-10m/m", "lte" :"now" } } } }'  
> docinfo =\> true  
> schedule =\> "\* \* \* \* \*"  
> }  
> }

I have another config that uses jdbc plugin which is pulling more than 10Gb of logs daily from SQL server to the same Elastic cluster and has no performance issues.

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [September 25, 2018, 9:54pm UTC](https://discuss.elastic.co/t/data-replication-from-one-es-to-another-using-logstash/149690/4 "2018-09-25T21:54:22Z")

</div>

Your initial problem statement indicates performance degradation and a possible node crash on what you are calling your destination server:

> [@Dolph\_2709](#):
>
> after few days when indexes close to 3 mil records, it starts to "slowdown" Kibana on destination server and even stopped ES instance.

I'm not sure how that is leading you to believe that the Logstash input for this pipeline is at fault. What am I missing?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 23, 2018, 9:54pm UTC](https://discuss.elastic.co/t/data-replication-from-one-es-to-another-using-logstash/149690/5 "2018-10-23T21:54:28Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.