Reindex to change type with wildcard indexes

Elastic Stack Logstash
1

Hello,

When I started to send data from logstash to elasticsearch the format was set to string for several inputs.
I'd like to change the format from string to integer, but it looks like it requires reindexing the whole stuff.

So I started to search how to reindex my logstash-* to logstashnew-* and change the format of some data from string to integer. (for the fields labelled "crsscore", "sentbyte", "rcvdbyte", "sentpkt" and "rcvdpkt").

All the examples I found so far use logstash with a single index.

Is it possible to process all indexes using a wildcard in the logstash input to be able to convert the format from string to integer when creating the new one ?

input {
  elasticsearch {
   hosts => "elasticsearch:9200"
   index => "logstash-*"
   size => 10
  }
}
filter {
  convert  {
       "sentbyte" =>"integer"
       "sentpkt" =>"integer"
       "rcvdbyte" =>"integer"
       "rcvdpkt" =>"integer"
       "crsscore" =>"integer"
 }
}
output {
 elasticsearch {
   host => "elasticsearch.net"
   port => 9200
   protocol => "http"
   manage_template => false
   index => "logconvert-"
   document_id => "%{id}"
   workers => 1
 }
}

Best regards.

2

Hey,

this might be more of a logstash and not an Elasticsearch question.

You might want to check the docinfo configuration in the Elasticsearch input, it even contains an example.

--Alex

3

Thanks for the hint, but I still have an error with docinfo.
08:42:12.345 [[main]<elasticsearch] ERROR logstash.pipeline - A plugin had an unrecoverable error. Will restart this plugin.
Plugin: <LogStash::Inputs::Elasticsearch hosts=>["elasticsearch-srv.europa.net"], index=>"logstash-2017.02.02", query=>"*", docinfo=>true, id=>"d0a8a12677067754d1a5d6720046586bafd939db-1", enable_metric=>true, codec=><LogStash::Codecs::JSON id=>"json_eef2e278-5651-4e5f-9dce-aa96ff2a7137", enable_metric=>true, charset=>"UTF-8">, size=>1000, scroll=>"1m", docinfo_target=>"@metadata", docinfo_fields=>["_index", "_type", "_id"], ssl=>false>
Error: [400] {"error":{"root_cause":[{"type":"parse_exception","reason":"Failed to derive xcontent"}],"type":"parse_exception","reason":"Failed to derive xcontent"},"status":400}

I'm still stuck with my reindexing attempt.

Best regards.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.