# Copy docs from one index to another index **URL:** \ **Category:** Elasticsearch\ **Created:** [November 12, 2015, 1:05pm UTC](https://discuss.elastic.co/t/copy-docs-from-one-index-to-another-index/34415 "2015-11-12T13:05:17Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![r.ganeshbabu](https://avatars.discourse-cdn.com/v4/letter/r/4da419/32.png) [@r.ganeshbabu](https://discuss.elastic.co/u/r.ganeshbabu)\ **Post date:** [November 12, 2015, 1:05pm UTC](https://discuss.elastic.co/t/copy-docs-from-one-index-to-another-index/34415/1 "2015-11-12T13:05:17Z")
Hi Team, I have two indexes(lets say es\_item, es\_item1) with same mapping and same index type. es\_item index contains 1000 docs and es\_item1 index contains 2000 docs. I want to copy or move docs from es\_item to es\_item1 index. Is it possible to do that? Thanks, Ganeshbabu R ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [November 12, 2015, 1:24pm UTC](https://discuss.elastic.co/t/copy-docs-from-one-index-to-another-index/34415/2 "2015-11-12T13:24:45Z")
Right now there isn't anything built especially for that. You'd have to scroll from one index and bulk into the other index. ---
**Author:** ![r.ganeshbabu](https://avatars.discourse-cdn.com/v4/letter/r/4da419/32.png) [@r.ganeshbabu](https://discuss.elastic.co/u/r.ganeshbabu)\ **Post date:** [November 12, 2015, 1:54pm UTC](https://discuss.elastic.co/t/copy-docs-from-one-index-to-another-index/34415/3 "2015-11-12T13:54:06Z")
Thanks for your response @nik9000 I tried to use scan and scroll for the reindex and I executed the below command to get scroll id GET /es\_item/\_search?search\_type=scan&scroll=1m { "query": { "match\_all": {}}, "size": 1000 } I got the below response { "\_scroll\_id": "c2Nhbjs1Ozc3MzE6R3FRUUxURGpUUE9rdjQxaGZoUWJBQTs3NzQwOlJoUWhVNWxIVFpXOV8xTmpLUGR6NUE7NzczMjpHcVFRTFREalRQT2ZmhRYkFBOzc3NDE6UmhRaFU1bEhUWlc5XzFOaktQZHo1QTs3NzMzOkdxUVFMVERqVFBPa3Y0MWhmaFFiQUE7MTt0b3RhbF9oaXRzOjEyMTY1Njs=", "took": 2, "timed\_out": false, "\_shards": { "total": 5, "successful": 5, "failed": 0 }, "hits": { "total": 121656, "max\_score": 0, "hits": [] } } I am new to the Elasticsearch scan & scroll. Can you tell me how to do bulk index by using the scroll id? Please let me know any sample code. Thanks, Ganeshbabu R ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 12, 2015, 2:14pm UTC](https://discuss.elastic.co/t/copy-docs-from-one-index-to-another-index/34415/4 "2015-11-12T14:14:32Z")
I also wrote a blog a post about this subject: [http://david.pilato.fr/blog/2015/05/20/reindex-elasticsearch-with-logstash/](http://david.pilato.fr/blog/2015/05/20/reindex-elasticsearch-with-logstash/) ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [November 12, 2015, 2:22pm UTC](https://discuss.elastic.co/t/copy-docs-from-one-index-to-another-index/34415/5 "2015-11-12T14:22:03Z")
> [@dadoonet](#): > > I also wrote a blog a post about this subject: [https://david.pilato.fr/blog/2015-05-20-reindex-elasticsearch-with-logstash/](http://david.pilato.fr/blog/2015/05/20/reindex-elasticsearch-with-logstash/) I was going to suggest reading the [perl](https://github.com/elastic/elasticsearch-perl/blob/6103b25b32d8d8c05e6f70872a3319067b52d754/lib/Search/Elasticsearch/Bulk.pm) client because I know it has a reindex. The logstash is likely to be less complicated though. ---
**Author:** ![r.ganeshbabu](https://avatars.discourse-cdn.com/v4/letter/r/4da419/32.png) [@r.ganeshbabu](https://discuss.elastic.co/u/r.ganeshbabu)\ **Post date:** [November 12, 2015, 4:58pm UTC](https://discuss.elastic.co/t/copy-docs-from-one-index-to-another-index/34415/6 "2015-11-12T16:58:51Z")
Thanks for your response @dadoonet ---
**Author:** ![r.ganeshbabu](https://avatars.discourse-cdn.com/v4/letter/r/4da419/32.png) [@r.ganeshbabu](https://discuss.elastic.co/u/r.ganeshbabu)\ **Post date:** [November 12, 2015, 4:59pm UTC](https://discuss.elastic.co/t/copy-docs-from-one-index-to-another-index/34415/7 "2015-11-12T16:59:15Z")
Thanks for your response @nik9000 ---
**Author:** ![r.ganeshbabu](https://avatars.discourse-cdn.com/v4/letter/r/4da419/32.png) [@r.ganeshbabu](https://discuss.elastic.co/u/r.ganeshbabu)\ **Post date:** [November 13, 2015, 12:31pm UTC](https://discuss.elastic.co/t/copy-docs-from-one-index-to-another-index/34415/8 "2015-11-13T12:31:47Z")
Hi David, I tried to copy docs from one index to another index by using logstash Below is the logstash.conf file I created. input { # We read from the "old" index elasticsearch { hosts =\> ["10.7.148.21"] port =\> "9200" index =\> "es\_item" user =\> "esadmin" password =\> "password" size =\> 500 scroll =\> "5m" docinfo =\> true } } output { # We write to the "new" index elasticsearch { host =\> "10.7.148.21" port =\> "9200" protocol =\> "http" user =\> "esadmin" password =\> "password" index =\> "es\_item1" index\_type =\> "item" document\_id =\> "16636281" } # We print dots to see it in action stdout { codec =\> "dots" } } When I executed the below command **bin/logstash -f logstash.conf** I got the following response:- **You are using a deprecated config setting "index\_type" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=\>"index\_type", :plugin=\>\, :level=\>:warn}** Logstash startup completed **Exception in thread "Ruby-0-Thread-7: /opt/esadmin/elasticsearch/bin/logstash-1.5.0/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/buffer.rb:92" java.lang.UnsupportedOperationException** **at java.lang.Thread.stop(Thread.java:869)** **at org.jruby.RubyThread.exceptionRaised(RubyThread.java:1221)** **at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:112)** **at java.lang.Thread.run(Thread.java:745)** Is this right way of doing the logstash config? Please guide me to fix this issue Thanks, Ganeshbabu R ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 13, 2015, 2:01pm UTC](https://discuss.elastic.co/t/copy-docs-from-one-index-to-another-index/34415/9 "2015-11-13T14:01:40Z")
First guess: add a empty filter section. If it does not work can you open a new thread in logstash group? And if it works, I would open an issue in logstash project. ---
**Author:** ![r.ganeshbabu](https://avatars.discourse-cdn.com/v4/letter/r/4da419/32.png) [@r.ganeshbabu](https://discuss.elastic.co/u/r.ganeshbabu)\ **Post date:** [November 14, 2015, 8:04am UTC](https://discuss.elastic.co/t/copy-docs-from-one-index-to-another-index/34415/10 "2015-11-14T08:04:35Z")
David, @dadoonet I tried by giving empty filter section but still I am facing the same error. Can you provide me logstash group link? so that I can open new thread. I am using elasticsearch 1.7.2 but I downloaded logstash 1.5.0 will it be the reason for getting this exception? **Note:- I see the docs count increase in the target index but due to the below exception entire docs are not copied to target index** **Exception in thread "Ruby-0-Thread-7: /opt/esadmin/elasticsearch/bin/logstash-1.5.0/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/buffer.rb:92" java.lang.UnsupportedOperationException** **at java.lang.Thread.stop(Thread.java:869)** **at org.jruby.RubyThread.exceptionRaised(RubyThread.java:1221)** **at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:112)** **at java.lang.Thread.run(Thread.java:745)** **Exception in thread "\ **Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 14, 2015, 8:23am UTC](https://discuss.elastic.co/t/copy-docs-from-one-index-to-another-index/34415/11 "2015-11-14T08:23:23Z") > **[Logstash](https://discuss.elastic.co/c/logstash)** > > Everything related to your favorite centralized logging platform, including plugins and recipes. ---
**Author:** ![r.ganeshbabu](https://avatars.discourse-cdn.com/v4/letter/r/4da419/32.png) [@r.ganeshbabu](https://discuss.elastic.co/u/r.ganeshbabu)\ **Post date:** [November 14, 2015, 8:25am UTC](https://discuss.elastic.co/t/copy-docs-from-one-index-to-another-index/34415/12 "2015-11-14T08:25:02Z")
Thanks for your quick response @dadoonet ---
**Author:** ![LeeDr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leedr/32/9289_2.png) [@LeeDr](https://discuss.elastic.co/u/LeeDr)\ **Post date:** [November 20, 2015, 3:24pm UTC](https://discuss.elastic.co/t/copy-docs-from-one-index-to-another-index/34415/13 "2015-11-20T15:24:38Z")
I'm an es newbie, but I used [https://github.com/taskrabbit/elasticsearch-dump](https://github.com/taskrabbit/elasticsearch-dump) to export some test data and re-import it. It might be useful to you. ---
**Author:** ![r.ganeshbabu](https://avatars.discourse-cdn.com/v4/letter/r/4da419/32.png) [@r.ganeshbabu](https://discuss.elastic.co/u/r.ganeshbabu)\ **Post date:** [November 20, 2015, 6:16pm UTC](https://discuss.elastic.co/t/copy-docs-from-one-index-to-another-index/34415/14 "2015-11-20T18:16:39Z")
Thanks for your response @LeeDr Sure I will do some testing in DEV environment and let you know the feedback. ---
**Author:** ![r.ganeshbabu](https://avatars.discourse-cdn.com/v4/letter/r/4da419/32.png) [@r.ganeshbabu](https://discuss.elastic.co/u/r.ganeshbabu)\ **Post date:** [December 11, 2015, 8:07am UTC](https://discuss.elastic.co/t/copy-docs-from-one-index-to-another-index/34415/15 "2015-12-11T08:07:13Z")
Hi @dadoonet, I upgraded to ES version from 1.7.2 to 1.7.3 & installed logstash 1.5.5 and trying to copy docs from one index to another index. Below is the sample\*\* logstash.conf\*\* file input { elasticsearch { hosts =\> ["10.7.148.21:9200"] user =\> "esadmin" password =\> "password" index =\> "item\_demo" size =\> 500 scroll =\> "5m" docinfo=\> true } } filter { } output { elasticsearch { host =\> "10.7.148.21:9200" protocol =\> "http" user =\> "esadmin" password =\> "password" index =\> "item\_test" } stdout { codec =\> "dots" } } When I executed the command bin/logstash -f logstash.conf I got the following error response:- failed action with response of 404, dropping action: ["index", {:id=\>nil, :index=\>"item\_demo\_test", :type=\>"logs", :routing=\>nil}, #@metadata\_accessors=#@store={"index"=\>"itemdemo", "type"=\>"item", "id"=\>"9151301", "retry\_count"=\>0}, @lut={}\>, @cancelled=false, @data={"ITEM\_ID"=\>9151301, "ITEM\_CODE"=\>"9151301", "ITEM\_DSCR"=\>"KITC HEN & TABLEWARE", "ITEM\_SPECIFICITY\_REF\_ID"=\>186, "ITEM\_TYPE"=\>"SGI", "IS\_SHARED\_IND"=\>"Y", "HAS\_IMAGE\_IND"=\>"Y", "HAS\_HIST\_IND"=\>"Y", "MOD\_CHR\_VAL\_ID"=\>12574160, "MOD\_DSCR"=\>"KITCHEN & TABLEWARE", "SG\_CHR\_VAL\_ID"=\>5439993, "PG\_CHR\_VAL\_ID"=\>17428187, "ITEM\_EU\_NAN\_CODE "=\>"8378994558", "ITEM\_EU\_NAN\_KEY"=\>"4422653", "ITEM\_CA\_CNAN"=\>nil, "ITEM\_US\_PRDC\_ID"=\>nil, "RELATIONSHIP"=\>[{"REL\_TYP\_REF\_ID"=\>-999, "REL\_TYPE"=\>"NO RELATIONSHIP", "REL\_CTGRY"=\>"NIL"}], "ITEM\_MISUSED\_GTIN\_FLG"=\>"N", "CRT\_DTTM"=\>"2004-09-25 22:00:00", "UPD\_DTTM"=\>"201 4-06-01 04:01:30", "DIST"=\>[{"RGN\_ID"=\>4, "RGN\_NM"=\>"DE", "DSTN\_STRT\_DT"=\>"2007-08-22 20:24:28", "DSTN\_END\_DT"=\>"9999-12-31 00:00:00", "ITEM\_GLBL\_CODE\_ST\_REF\_ID"=\>720, "ITEM\_LCL\_CODE\_ST\_REF\_ID"=\>720, "IS\_FFU\_IND"=\>"Y", "FIRST\_FFU\_DT"=\>"2012-07-06 16:17:18", "HAS\_LPV\_I ND"=\>"N", "FIRST\_DATA\_DT"=\>nil, "LAST\_DATA\_DT"=\>nil, "HAS\_HIST\_IND"=\>"Y", "LAST\_CHG\_DT"=\>"2004-09-26 10:02:29", "IS\_PREMVMNT\_IND"=\>"N" , "PRE\_MOVEMENT\_DT"=\>nil} Logstash 1.5.5 is worked well for ES version 1.7.2. Please guide me to resolve this issue it would be very helpful. Thanks, Ganeshbabu R ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [December 11, 2015, 8:35am UTC](https://discuss.elastic.co/t/copy-docs-from-one-index-to-another-index/34415/16 "2015-12-11T08:35:49Z")
I'd open another thread in logstash forum. But apparently you have a 404 error which means that your index does not exist. Check `GET 10.7.148.21:9200/item_demo_test` ---
**Author:** ![r.ganeshbabu](https://avatars.discourse-cdn.com/v4/letter/r/4da419/32.png) [@r.ganeshbabu](https://discuss.elastic.co/u/r.ganeshbabu)\ **Post date:** [December 11, 2015, 8:56am UTC](https://discuss.elastic.co/t/copy-docs-from-one-index-to-another-index/34415/17 "2015-12-11T08:56:32Z")
Thanks for your response @dadoonet I am getting the response using curl GET command [esadmin@dayrhezhmd001 elasticsearch-1.7.3]$ curl --user esadmin:password -XGET '[http://10.7.148.21:9200/item\_demo\_test/?pretty=true](http://10.7.148.21:9200/item_demo_test/?pretty=true)' **Sample:-** { "item\_demo\_test" : { "aliases" : { }, "mappings" : { "item" : { "\_all" : { "enabled" : false }, "properties" : { "CRT\_DTTM" : { "type" : "date", "doc\_values" : true, "format" : "yyyy-MM-dd HH:mm:ss" }, "DIST" : { "properties" : { "DSTN\_END\_DT" : { "type" : "date", "doc\_values" : true, "format" : "yyyy-MM-dd HH:mm:ss" }, "DSTN\_STRT\_DT" : { "type" : "date", "doc\_values" : true, "format" : "yyyy-MM-dd HH:mm:ss" }, } } } } Let me know your feedback. Thanks, Ganeshbabu R ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [December 11, 2015, 9:21am UTC](https://discuss.elastic.co/t/copy-docs-from-one-index-to-another-index/34415/18 "2015-12-11T09:21:38Z")
I have no idea. I'd open this discussion in logstash group. ---
**Author:** ![r.ganeshbabu](https://avatars.discourse-cdn.com/v4/letter/r/4da419/32.png) [@r.ganeshbabu](https://discuss.elastic.co/u/r.ganeshbabu)\ **Post date:** [February 11, 2016, 2:25pm UTC](https://discuss.elastic.co/t/copy-docs-from-one-index-to-another-index/34415/19 "2016-02-11T14:25:09Z")
Hi @dadoonet I tried copying docs from old index to new index. I executed the logstash run command using **nohup** and running the logstash in background make docs count keep on increasing. **Below is the logstash conf file,** input { # We read from the "old" index elasticsearch { hosts =\> ["10.8.42.121:9200"] user =\> "esadmin" password =\> "Gemmot08" index =\> "es\_item" size =\> 1000 scroll =\> "1m" docinfo=\> true } } filter { } output { # We write to the "new" index elasticsearch { host =\> "10.8.42.121:9200" protocol =\> "http" user =\> "esadmin" password =\> "xxxxx" index =\> "es\_item2" document\_type =\> "item" } # We print dots to see it in action stdout { codec =\> "dots" } } **Logstash run command** nohup ./bin/logstash -f logstash.conf & Logstash never stops when running in background process and keeps on adding docs to new index and it causing large index size. How to terminate the logstash once docs copy is completed? Is there any other way to run logstash in background? Thanks, Ganeshbabu R ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 11, 2016, 3:26pm UTC](https://discuss.elastic.co/t/copy-docs-from-one-index-to-another-index/34415/20 "2016-02-11T15:26:50Z")
Kill -9 ? Why running with nohup and &? [Next page](https://discuss.elastic.co/t/copy-docs-from-one-index-to-another-index/34415.md?page=2)