Using logstash to reindex Elasticsearch

Elastic Stack Logstash
1

I recently read DavidPilato's blog post : http://david.pilato.fr/blog/2015/05/20/reindex-elasticsearch-with-logstash/
Its a cool thing you can do, and for the most part it works just like David mentioned except for some problems :

  1. It does not seem to work with creating the index based on a template. (I would like to copy the content from instance1_index1 and write it to instance2_index2(which is based on a template))

  2. If I provide a type, it issues a warning saying the index_type declaration is deprecated, and then proceeds to spew out errors :

    failed action with response of 400, dropping action: ["index", {:_id=>""cf4695d5-100c-4c8a-89f5-6c91e9ba01c8"", :_index=>"blip-2015.09.15", :_type=>"logs", :_routing=>nil}, #<LogStash::Event:0x1c82492f @metadata={"_index"=>"blip-2015.09.15", "_type"=>"twitter", "_id"=>""cf4695d5-100c-4c8a-89f5-6c91e9ba01c8"", "retry_count"=> .....

here is my config ;

input {
   elasticsearch {
     hosts => ["porto.int.thomsonreuters.com"]
     port => "7200"
     index => "blip-2015.09.15"
     scroll => "5m"
     docinfo => true
     query => '{"query": {"filtered": {"filter": {"exists": {"field": "raw.place.bounding_box.coordinates"}}}}}'
   }
}

filter {
  mutate {
    remove_field => ["atimestamp","@version"]
    update => {"raw.place.bounding_box.type" => "multipoint" }
  }
}

output {
   elasticsearch {
     host => "localhost"
     port => 9200
     protocol => "http"
     index => "%{[@metadata][_index]}"
     #index_type => "twitter"
     document_id => "%{[@metadata][_id]}"
   }
   stdout {
     codec => "dots"
   }
}

Any ideas on if this can be made to work with index-templates ?

Thanks

Ramdev

2

I believe the docs say that index_type has been deprecated in favor of document_type, so replacing index_type with document_type should fix that problem. :smile: