# Script to move data between clusters
**URL:** https://discuss.elastic.co/t/script-to-move-data-between-clusters/3557
**Category:** Elasticsearch
**Created:** [November 11, 2010, 5:39pm UTC](https://discuss.elastic.co/t/script-to-move-data-between-clusters/3557 "2010-11-11T17:39:14Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@ppearcy](https://discuss.elastic.co/u/ppearcy)
#### Post date: [November 11, 2010, 5:39pm UTC](https://discuss.elastic.co/t/script-to-move-data-between-clusters/3557/1 "2010-11-11T17:39:14Z")
Hey,
Was curious if anybody had already written scripts to copy data
between clusters by pulling docs from one cluster and indexing to
another. Preferably, something using the bulk APIs.
I started to write this against the java node client until I came to
the realization that this would not work between different versions,
as my app must choose to be compatible with 0.12 based cluster or a
0.13 based cluster depending on which ES jar I am pulling in. This is
the one minor draw back of using the Node client. I am not a java
expert, so maybe there is some way to pull in both jar files into a
single app(my only thought is that I could hack one to have a slightly
different import path, but that seems very sketchy). The other
alternative in Java is two apps, one to do a data dump and the other
to import these files.
Currently, we are just re-flowing data from our backend storage, but
our backend data store is not that fast and there is a good amount of
overhead extracting content from docs (especially PDFs).
Thanks!
Paul
---
### Author:  [@Ryan\_Crumley](https://discuss.elastic.co/u/Ryan_Crumley)
#### Post date: [November 11, 2010, 5:46pm UTC](https://discuss.elastic.co/t/script-to-move-data-between-clusters/3557/2 "2010-11-11T17:46:46Z")
You could use osgi to load both versions into the same jvm. Here is an
example that loads multiple groovy versions into the same jvm:
> **[Beginner's Guide to OSGi on the Desktop](http://hamletdarcy.blogspot.com/2008/12/beginners-guide-to-osgi-on-desktop.html)**
>
> ... a beginner's guide to OSGi on the desktop in which none of the following are discussed: history, politics, the spec, mobile devices, Su...
OSGI itself can be painful... the tooling often does not work as well as you
hope.
[http://hamletdarcy.blogspot.com/2008/12/beginners-guide-to-osgi-on-desktop.html](http://hamletdarcy.blogspot.com/2008/12/beginners-guide-to-osgi-on-desktop.html)
Ryan
On Thu, Nov 11, 2010 at 11:39 AM, Paul [ppearcy@gmail.com](mailto:ppearcy@gmail.com) wrote:
> Hey,
> Was curious if anybody had already written scripts to copy data
> between clusters by pulling docs from one cluster and indexing to
> another. Preferably, something using the bulk APIs.
>
> I started to write this against the java node client until I came to
> the realization that this would not work between different versions,
> as my app must choose to be compatible with 0.12 based cluster or a
> 0.13 based cluster depending on which ES jar I am pulling in. This is
> the one minor draw back of using the Node client. I am not a java
> expert, so maybe there is some way to pull in both jar files into a
> single app(my only thought is that I could hack one to have a slightly
> different import path, but that seems very sketchy). The other
> alternative in Java is two apps, one to do a data dump and the other
> to import these files.
>
> Currently, we are just re-flowing data from our backend storage, but
> our backend data store is not that fast and there is a good amount of
> overhead extracting content from docs (especially PDFs).
>
> Thanks!
> Paul
---
### Author:  [@ppearcy](https://discuss.elastic.co/u/ppearcy)
#### Post date: [November 11, 2010, 5:57pm UTC](https://discuss.elastic.co/t/script-to-move-data-between-clusters/3557/3 "2010-11-11T17:57:29Z")
Cool, thanks Ryan. Will take a look at that approach.
On Nov 11, 10:46 am, Ryan Crumley [crum...@gmail.com](mailto:crum...@gmail.com) wrote:
> You could use osgi to load both versions into the same jvm. Here is an
> example that loads multiple groovy versions into the same jvm:
>
> [http://hamletdarcy.blogspot.com/2008/12/beginners-guide-to-osgi-on-de](http://hamletdarcy.blogspot.com/2008/12/beginners-guide-to-osgi-on-de)...
>
> OSGI itself can be painful... the tooling often does not work as well as you
> hope.
>
> [http://hamletdarcy.blogspot.com/2008/12/beginners-guide-to-osgi-on-de...](http://hamletdarcy.blogspot.com/2008/12/beginners-guide-to-osgi-on-de...)
> Ryan
>
> On Thu, Nov 11, 2010 at 11:39 AM, Paul [ppea...@gmail.com](mailto:ppea...@gmail.com) wrote:
>
> > Hey,
> > Was curious if anybody had already written scripts to copy data
> > between clusters by pulling docs from one cluster and indexing to
> > another. Preferably, something using the bulk APIs.
>
> > I started to write this against the java node client until I came to
> > the realization that this would not work between different versions,
> > as my app must choose to be compatible with 0.12 based cluster or a
> > 0.13 based cluster depending on which ES jar I am pulling in. This is
> > the one minor draw back of using the Node client. I am not a java
> > expert, so maybe there is some way to pull in both jar files into a
> > single app(my only thought is that I could hack one to have a slightly
> > different import path, but that seems very sketchy). The other
> > alternative in Java is two apps, one to do a data dump and the other
> > to import these files.
>
> > Currently, we are just re-flowing data from our backend storage, but
> > our backend data store is not that fast and there is a good amount of
> > overhead extracting content from docs (especially PDFs).
>
> > Thanks!
> > Paul
---
### Author:  [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)
#### Post date: [November 11, 2010, 7:13pm UTC](https://discuss.elastic.co/t/script-to-move-data-between-clusters/3557/4 "2010-11-11T19:13:52Z")
Hi Paul
> Was curious if anybody had already written scripts to copy data
> between clusters by pulling docs from one cluster and indexing to
> another. Preferably, something using the bulk APIs.
Here's a simple Perl script which will do what you want in parallel
(using fork).
Should be fairly self explanatory. You would need to install these Perl
modules:
- Elasticsearch.pm:
[ElasticSearch-0.68 - DEPRECATED: An API for communicating with ElasticSearch - metacpan.org](http://search.cpan.org/~drtech/ElasticSearch/)
- Parallel::ForkManager:
[Parallel-ForkManager-0.7.9 - A simple parallel processing fork manager - metacpan.org](http://search.cpan.org/~dlux/Parallel-ForkManager/)
#!/usr/bin/perl
use strict;
use warnings;
use Elasticsearch();
use Parallel::ForkManager();
use Data::Dumper;
our $Source\_ES = '127.0.0.1:9200';
our $Dest\_ES = '127.0.0.1:9201';
our $Source\_Index = 'my\_index';
our $Dest\_Index = $Source\_Index;
our $Max\_Kids = 10;
our $Rows = 1000;
our $Per\_Kid = 10 \* $Rows;
my $source = Elasticsearch-\>new( servers =\> $Source\_ES );
my $dest = Elasticsearch-\>new( servers =\> $Dest\_ES );
my $pm = Parallel::ForkManager-\>new($Max\_Kids);
$|++; # Auto-flush STDOUT to see progress
main();
#===================================
sub main {
#===================================
delete\_index();
create\_index();
put\_mappings();
my $total = total\_docs();
my $start = 0;
while ( $start \< $total ) {
my $end = $start + $Per\_Kid;
if ( $pm-\>start ) {
```
# parent
$start = $end;
next;
}
# child
index_docs( $start, $end );
$pm->finish;
}
$pm->wait_all_children;
print "\n - Done - \n";
```
}
#===================================
sub delete\_index {
#===================================
print "Deleting index '$Dest\_Index' in case it already exists\n";
eval {
$dest-\>delete\_index( index =\> $Dest\_Index );
wait\_for\_es();
};
}
#===================================
sub create\_index {
#===================================
print "Creating index '$Dest\_Index'\n";
$dest-\>create\_index( index =\> $Dest\_Index );
wait\_for\_es();
}
#===================================
sub put\_mappings {
#===================================
print "Adding mappings\n";
my ($mappings) = values %{ $source-\>mapping( index =\> $Source\_Index ) };
for my $type ( sort keys %$mappings ) {
print " - $type\n";
my $mapping = $mappings-\>{$type};
$dest-\>put\_mapping(
index =\> $Dest\_Index,
type =\> $type,
map { $\_ =\> $mapping-\>{$\_} } qw(\_all \_source properties)
);
}
wait\_for\_es();
}
#===================================
sub total\_docs {
#===================================
my $total = $source-\>count(
index =\> $Source\_Index,
match\_all =\> {}
)-\>{count};
```
print "Indexing $total docs from '$Source_Index' to '$Dest_Index\n";
return $total;
```
}
#===================================
sub index\_docs {
#===================================
my $start = shift;
my $end = shift;
```
while ( $start < $end ) {
print ".";
my @objects = map {
create => {
index => $Dest_Index,
type => $_->{_type},
id => $_->{_id},
data => $_->{_source}
}
},
@{
$source->search(
index => $Source_Index,
query => { match_all => {} },
sort => ['_id'],
from => $start,
size => $Rows
)->{hits}{hits}
};
my $result = $dest->bulk( \@objects );
die Dumper($result) if $result->{errors};
last if @objects < $Rows;
$start += $Rows;
}
```
}
#===================================
sub wait\_for\_es {
#===================================
$dest-\>cluster\_health( wait\_for\_status =\> 'yellow' );
}
clint
---
### Author:  [@ppearcy](https://discuss.elastic.co/u/ppearcy)
#### Post date: [November 12, 2010, 1:04am UTC](https://discuss.elastic.co/t/script-to-move-data-between-clusters/3557/5 "2010-11-12T01:04:33Z")
Awesome, thanks! Very helpful.
On Nov 11, 12:13 pm, Clinton Gormley [clin...@iannounce.co.uk](mailto:clin...@iannounce.co.uk) wrote:
> Hi Paul
>
> > Was curious if anybody had already written scripts to copy data
> > between clusters by pulling docs from one cluster and indexing to
> > another. Preferably, something using the bulk APIs.
>
> Here's a simple Perl script which will do what you want in parallel
> (using fork).
>
> Should be fairly self explanatory. You would need to install these Perl
> modules:
>
> - Elasticsearch.pm:
> [ElasticSearch-0.68 - DEPRECATED: An API for communicating with ElasticSearch - metacpan.org](http://search.cpan.org/~drtech/ElasticSearch/)
> - Parallel::ForkManager:
> [Parallel-ForkManager-0.7.9 - A simple parallel processing fork manager - metacpan.org](http://search.cpan.org/~dlux/Parallel-ForkManager/)
>
> #!/usr/bin/perl
>
> use strict;
> use warnings;
> use Elasticsearch();
> use Parallel::ForkManager();
>
> use Data::Dumper;
>
> our $Source\_ES = '127.0.0.1:9200';
> our $Dest\_ES = '127.0.0.1:9201';
> our $Source\_Index = 'my\_index';
> our $Dest\_Index = $Source\_Index;
> our $Max\_Kids = 10;
> our $Rows = 1000;
> our $Per\_Kid = 10 \* $Rows;
>
> my $source = Elasticsearch-\>new( servers =\> $Source\_ES );
> my $dest = Elasticsearch-\>new( servers =\> $Dest\_ES );
> my $pm = Parallel::ForkManager-\>new($Max\_Kids);
>
> $|++; # Auto-flush STDOUT to see progress
>
> main();
>
> #===================================
> sub main {
> #===================================
> delete\_index();
> create\_index();
> put\_mappings();
> my $total = total\_docs();
> my $start = 0;
> while ( $start \< $total ) {
> my $end = $start + $Per\_Kid;
> if ( $pm-\>start ) {
>
> ```
> # parent
> $start = $end;
> next;
> }
>
> # child
> index_docs( $start, $end );
> $pm->finish;
>
> }
> $pm->wait_all_children;
> print "\n - Done - \n";
>
> ```
>
> }
>
> #===================================
> sub delete\_index {
> #===================================
> print "Deleting index '$Dest\_Index' in case it already exists\n";
> eval {
> $dest-\>delete\_index( index =\> $Dest\_Index );
> wait\_for\_es();
> };
>
> }
>
> #===================================
> sub create\_index {
> #===================================
> print "Creating index '$Dest\_Index'\n";
> $dest-\>create\_index( index =\> $Dest\_Index );
> wait\_for\_es();
>
> }
>
> #===================================
> sub put\_mappings {
> #===================================
> print "Adding mappings\n";
> my ($mappings) = values %{ $source-\>mapping( index =\> $Source\_Index ) };
> for my $type ( sort keys %$mappings ) {
> print " - $type\n";
> my $mapping = $mappings-\>{$type};
> $dest-\>put\_mapping(
> index =\> $Dest\_Index,
> type =\> $type,
> map { $\_ =\> $mapping-\>{$\_} } qw(\_all \_source properties)
> );
> }
> wait\_for\_es();
>
> }
>
> #===================================
> sub total\_docs {
> #===================================
> my $total = $source-\>count(
> index =\> $Source\_Index,
> match\_all =\> {}
> )-\>{count};
>
> ```
> print "Indexing $total docs from '$Source_Index' to '$Dest_Index\n";
> return $total;
>
> ```
>
> }
>
> #===================================
> sub index\_docs {
> #===================================
> my $start = shift;
> my $end = shift;
>
> ```
> while ( $start < $end ) {
> print ".";
> my @objects = map {
> create => {
> index => $Dest_Index,
> type => $_->{_type},
> id => $_->{_id},
> data => $_->{_source}
> }
> },
> @{
> $source->search(
> index => $Source_Index,
> query => { match_all => {} },
> sort => ['_id'],
> from => $start,
> size => $Rows
> )->{hits}{hits}
> };
>
> my $result = $dest->bulk( \@objects );
> die Dumper($result) if $result->{errors};
>
> last if @objects < $Rows;
> $start += $Rows;
> }
>
> ```
>
> }
>
> #===================================
> sub wait\_for\_es {
> #===================================
> $dest-\>cluster\_health( wait\_for\_status =\> 'yellow' );
>
> }
>
> clint
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [November 12, 2010, 5:57am UTC](https://discuss.elastic.co/t/script-to-move-data-between-clusters/3557/6 "2010-11-12T05:57:38Z")
Hi,
Yea, the way to go with that is using the REST based interface, which is
less likely to change its underlying format :). The perl client is
one possibility, another is by using HTTP in Java.
```
Actually, there could be a feature in elasticsearch where you would say:
```
Reindex "indexX" into this HTTP endpoint. Can be implemented and should be
faster.
-shay.banon
On Fri, Nov 12, 2010 at 3:04 AM, Paul [ppearcy@gmail.com](mailto:ppearcy@gmail.com) wrote:
> Awesome, thanks! Very helpful.
>
> On Nov 11, 12:13 pm, Clinton Gormley [clin...@iannounce.co.uk](mailto:clin...@iannounce.co.uk) wrote:
>
> > Hi Paul
> >
> > > Was curious if anybody had already written scripts to copy data
> > > between clusters by pulling docs from one cluster and indexing to
> > > another. Preferably, something using the bulk APIs.
> >
> > Here's a simple Perl script which will do what you want in parallel
> > (using fork).
> >
> > Should be fairly self explanatory. You would need to install these Perl
> > modules:
> >
> > - Elasticsearch.pm:
> > [ElasticSearch-0.68 - DEPRECATED: An API for communicating with ElasticSearch - metacpan.org](http://search.cpan.org/~drtech/ElasticSearch/)
> > - Parallel::ForkManager:
> > [Parallel-ForkManager-0.7.9 - A simple parallel processing fork manager - metacpan.org](http://search.cpan.org/~dlux/Parallel-ForkManager/)
> >
> > #!/usr/bin/perl
> >
> > use strict;
> > use warnings;
> > use Elasticsearch();
> > use Parallel::ForkManager();
> >
> > use Data::Dumper;
> >
> > our $Source\_ES = '127.0.0.1:9200';
> > our $Dest\_ES = '127.0.0.1:9201';
> > our $Source\_Index = 'my\_index';
> > our $Dest\_Index = $Source\_Index;
> > our $Max\_Kids = 10;
> > our $Rows = 1000;
> > our $Per\_Kid = 10 \* $Rows;
> >
> > my $source = Elasticsearch-\>new( servers =\> $Source\_ES );
> > my $dest = Elasticsearch-\>new( servers =\> $Dest\_ES );
> > my $pm = Parallel::ForkManager-\>new($Max\_Kids);
> >
> > $|++; # Auto-flush STDOUT to see progress
> >
> > main();
> >
> > #===================================
> > sub main {
> > #===================================
> > delete\_index();
> > create\_index();
> > put\_mappings();
> > my $total = total\_docs();
> > my $start = 0;
> > while ( $start \< $total ) {
> > my $end = $start + $Per\_Kid;
> > if ( $pm-\>start ) {
> >
> > ```
> > # parent
> > $start = $end;
> > next;
> > }
> >
> > # child
> > index_docs( $start, $end );
> > $pm->finish;
> >
> > }
> > $pm->wait_all_children;
> > print "\n - Done - \n";
> >
> > ```
> >
> > }
> >
> > #===================================
> > sub delete\_index {
> > #===================================
> > print "Deleting index '$Dest\_Index' in case it already exists\n";
> > eval {
> > $dest-\>delete\_index( index =\> $Dest\_Index );
> > wait\_for\_es();
> > };
> >
> > }
> >
> > #===================================
> > sub create\_index {
> > #===================================
> > print "Creating index '$Dest\_Index'\n";
> > $dest-\>create\_index( index =\> $Dest\_Index );
> > wait\_for\_es();
> >
> > }
> >
> > #===================================
> > sub put\_mappings {
> > #===================================
> > print "Adding mappings\n";
> > my ($mappings) = values %{ $source-\>mapping( index =\> $Source\_Index )
> > };
> > for my $type ( sort keys %$mappings ) {
> > print " - $type\n";
> > my $mapping = $mappings-\>{$type};
> > $dest-\>put\_mapping(
> > index =\> $Dest\_Index,
> > type =\> $type,
> > map { $\_ =\> $mapping-\>{$\_} } qw(\_all \_source properties)
> > );
> > }
> > wait\_for\_es();
> >
> > }
> >
> > #===================================
> > sub total\_docs {
> > #===================================
> > my $total = $source-\>count(
> > index =\> $Source\_Index,
> > match\_all =\> {}
> > )-\>{count};
> >
> > ```
> > print "Indexing $total docs from '$Source_Index' to '$Dest_Index\n";
> > return $total;
> >
> > ```
> >
> > }
> >
> > #===================================
> > sub index\_docs {
> > #===================================
> > my $start = shift;
> > my $end = shift;
> >
> > ```
> > while ( $start < $end ) {
> > print ".";
> > my @objects = map {
> > create => {
> > index => $Dest_Index,
> > type => $_->{_type},
> > id => $_->{_id},
> > data => $_->{_source}
> > }
> > },
> > @{
> > $source->search(
> > index => $Source_Index,
> > query => { match_all => {} },
> > sort => ['_id'],
> > from => $start,
> > size => $Rows
> > )->{hits}{hits}
> > };
> >
> > my $result = $dest->bulk( \@objects );
> > die Dumper($result) if $result->{errors};
> >
> > last if @objects < $Rows;
> > $start += $Rows;
> > }
> >
> > ```
> >
> > }
> >
> > #===================================
> > sub wait\_for\_es {
> > #===================================
> > $dest-\>cluster\_health( wait\_for\_status =\> 'yellow' );
> >
> > }
> >
> > clint
---
### Author:  [@ppearcy](https://discuss.elastic.co/u/ppearcy)
#### Post date: [November 14, 2010, 1:17am UTC](https://discuss.elastic.co/t/script-to-move-data-between-clusters/3557/7 "2010-11-14T01:17:29Z")
+1 for that feature. Would provide a very easy upgrade path for
breaking changes in the gateway.
I opened a feature request:
>
>
> Probably not a high priority or must have feature, but this would be extremely c…onvenient.
>
> This would allow one to:
> \- Switch gateways
> \- Upgrade to new version with incompatible gateway
> \- Clone an entire cluster
>
> Based on this discussion here:
> http://groups.google.com/a/elasticsearch.com/group/users/browse\_thread/thread/43b534ca360d808f?pli=1
Thanks!
On Nov 11, 10:57 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> Hi,
>
> Yea, the way to go with that is using the REST based interface, which is
> less likely to change its underlying format :). The perl client is
> one possibility, another is by using HTTP in Java.
>
> ```
> Actually, there could be a feature in elasticsearch where you would say:
>
> ```
>
> Reindex "indexX" into this HTTP endpoint. Can be implemented and should be
> faster.
>
> -shay.banon
>
> On Fri, Nov 12, 2010 at 3:04 AM, Paul [ppea...@gmail.com](mailto:ppea...@gmail.com) wrote:
>
> > Awesome, thanks! Very helpful.
>
> > On Nov 11, 12:13 pm, Clinton Gormley [clin...@iannounce.co.uk](mailto:clin...@iannounce.co.uk) wrote:
> >
> > > Hi Paul
>
> > > > Was curious if anybody had already written scripts to copy data
> > > > between clusters by pulling docs from one cluster and indexing to
> > > > another. Preferably, something using the bulk APIs.
>
> > > Here's a simple Perl script which will do what you want in parallel
> > > (using fork).
>
> > > Should be fairly self explanatory. You would need to install these Perl
> > > modules:
>
> > > - Elasticsearch.pm:
> > > [ElasticSearch-0.68 - DEPRECATED: An API for communicating with ElasticSearch - metacpan.org](http://search.cpan.org/~drtech/ElasticSearch/)
> > > - Parallel::ForkManager:
> > > [Parallel-ForkManager-0.7.9 - A simple parallel processing fork manager - metacpan.org](http://search.cpan.org/~dlux/Parallel-ForkManager/)
>
> > > #!/usr/bin/perl
>
> > > use strict;
> > > use warnings;
> > > use Elasticsearch();
> > > use Parallel::ForkManager();
>
> > > use Data::Dumper;
>
> > > our $Source\_ES = '127.0.0.1:9200';
> > > our $Dest\_ES = '127.0.0.1:9201';
> > > our $Source\_Index = 'my\_index';
> > > our $Dest\_Index = $Source\_Index;
> > > our $Max\_Kids = 10;
> > > our $Rows = 1000;
> > > our $Per\_Kid = 10 \* $Rows;
>
> > > my $source = Elasticsearch-\>new( servers =\> $Source\_ES );
> > > my $dest = Elasticsearch-\>new( servers =\> $Dest\_ES );
> > > my $pm = Parallel::ForkManager-\>new($Max\_Kids);
>
> > > $|++; # Auto-flush STDOUT to see progress
>
> > > main();
>
> > > #===================================
> > > sub main {
> > > #===================================
> > > delete\_index();
> > > create\_index();
> > > put\_mappings();
> > > my $total = total\_docs();
> > > my $start = 0;
> > > while ( $start \< $total ) {
> > > my $end = $start + $Per\_Kid;
> > > if ( $pm-\>start ) {
>
> > > ```
> > > # parent
> > > $start = $end;
> > > next;
> > > }
> > >
> > > ```
>
> > > ```
> > > # child
> > > index_docs( $start, $end );
> > > $pm->finish;
> > >
> > > ```
>
> > > ```
> > > }
> > > $pm->wait_all_children;
> > > print "\n - Done - \n";
> > >
> > > ```
>
> > > }
>
> > > #===================================
> > > sub delete\_index {
> > > #===================================
> > > print "Deleting index '$Dest\_Index' in case it already exists\n";
> > > eval {
> > > $dest-\>delete\_index( index =\> $Dest\_Index );
> > > wait\_for\_es();
> > > };
>
> > > }
>
> > > #===================================
> > > sub create\_index {
> > > #===================================
> > > print "Creating index '$Dest\_Index'\n";
> > > $dest-\>create\_index( index =\> $Dest\_Index );
> > > wait\_for\_es();
>
> > > }
>
> > > #===================================
> > > sub put\_mappings {
> > > #===================================
> > > print "Adding mappings\n";
> > > my ($mappings) = values %{ $source-\>mapping( index =\> $Source\_Index )
> > > };
> > > for my $type ( sort keys %$mappings ) {
> > > print " - $type\n";
> > > my $mapping = $mappings-\>{$type};
> > > $dest-\>put\_mapping(
> > > index =\> $Dest\_Index,
> > > type =\> $type,
> > > map { $\_ =\> $mapping-\>{$\_} } qw(\_all \_source properties)
> > > );
> > > }
> > > wait\_for\_es();
>
> > > }
>
> > > #===================================
> > > sub total\_docs {
> > > #===================================
> > > my $total = $source-\>count(
> > > index =\> $Source\_Index,
> > > match\_all =\> {}
> > > )-\>{count};
>
> > > ```
> > > print "Indexing $total docs from '$Source_Index' to '$Dest_Index\n";
> > > return $total;
> > >
> > > ```
>
> > > }
>
> > > #===================================
> > > sub index\_docs {
> > > #===================================
> > > my $start = shift;
> > > my $end = shift;
>
> > > ```
> > > while ( $start < $end ) {
> > > print ".";
> > > my @objects = map {
> > > create => {
> > > index => $Dest_Index,
> > > type => $_->{_type},
> > > id => $_->{_id},
> > > data => $_->{_source}
> > > }
> > > },
> > > @{
> > > $source->search(
> > > index => $Source_Index,
> > > query => { match_all => {} },
> > > sort => ['_id'],
> > > from => $start,
> > > size => $Rows
> > > )->{hits}{hits}
> > > };
> > >
> > > ```
>
> > > ```
> > > my $result = $dest->bulk( \@objects );
> > > die Dumper($result) if $result->{errors};
> > >
> > > ```
>
> > > ```
> > > last if @objects < $Rows;
> > > $start += $Rows;
> > > }
> > >
> > > ```
>
> > > }
>
> > > #===================================
> > > sub wait\_for\_es {
> > > #===================================
> > > $dest-\>cluster\_health( wait\_for\_status =\> 'yellow' );
>
> > > }
>
> > > clint
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 4:16am UTC](https://discuss.elastic.co/t/script-to-move-data-between-clusters/3557/8 "2017-07-06T04:16:32Z")