# Reindexing with Logstash causes ArrayIndexOutOfBoundsException errors on Elasticsearch 1.7.5 **URL:** \ **Category:** Elasticsearch\ **Created:** [April 6, 2016, 5:09pm UTC](https://discuss.elastic.co/t/reindexing-with-logstash-causes-arrayindexoutofboundsexception-errors-on-elasticsearch-1-7-5/46574 "2016-04-06T17:09:09Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![jdoss](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jdoss/32/8611_2.png) [@jdoss](https://discuss.elastic.co/u/jdoss)\ **Post date:** [April 6, 2016, 5:09pm UTC](https://discuss.elastic.co/t/reindexing-with-logstash-causes-arrayindexoutofboundsexception-errors-on-elasticsearch-1-7-5/46574/1 "2016-04-06T17:09:09Z")
I am seeing the following errors when trying to reindex all of my 2016 indexes to get rid of fields with dots in them so I can upgrade to Elasticsearch 2.x. I am seeing ArrayIndexOutOfBoundsException errors if I do all of 2016 (`logstash-2016.*`) or a single month of indexes (`logstash-2016.01.*`) with logstash and Elasticsearch 1.7.5. Here are the two kinds of errors that are kicked out. ``` A plugin had an unrecoverable error. Will restart this plugin. Plugin: ["lame-02"], index=>"logstash-2016.*", size=>10, scroll=>"5m", docinfo=>true, scan=>true, codec=>"UTF-8">, query=>"{\"query\": { \"match_all\": {} } }", docinfo_target=>"@metadata", docinfo_fields=>["_index", "_type", "_i d"], ssl=>false> Error: [500] {"error":"ArrayIndexOutOfBoundsException[null]","status":500} {:level=>:error} A plugin had an unrecoverable error. Will restart this plugin. Plugin: ["lame-02"], index=>"logstash-2016.*", size=>10, scroll=>"5m", docinfo=>true, scan=>true, codec=>"UTF-8">, query=>"{\"query\": { \"match_all\": {} } }", docinfo_target=>"@metadata", docinfo_fields=>["_index", "_type", "_i d"], ssl=>false> Error: [500] {"error":"ArrayIndexOutOfBoundsException[-131071]","status":500} {:level=>:error} ``` and here is the config I am using with Logstash 2.2.2: ``` input { elasticsearch { hosts => "client-02" index => "logstash-2016.01.*" size => 1000 scroll => "5m" docinfo => true scan => true } } filter { metrics { meter => "events" add_tag => "metric" flush_interval => 10 } } output { if "metric" in [tags] { stdout { codec => line { format => "count: %{[events][count]} rate: %{[events][rate_1m]}" } } } else { elasticsearch { hosts => "localhost" index => "%{[@metadata][_index]}" document_type => "%{[@metadata][_type]}" document_id => "%{[@metadata][_id]}" flush_size => 250 idle_flush_time => 10 workers => 4 } } } ``` I was able to reindex `logstash-2015.*` which only had 66 total indexes and 179323407 documents with the same config without issue. The `logstash-2016.*` index total is 97 and has 262633593 documents. Trying a single month in 2016 (`logstash-2016.01.*)` which has 31 indexes and 81231923 documents results in the same kind of errors. I also tried dropping the size down from 1000 to 100 and the error persisted. I can reindex a single 2016 month at a time but that is going to be a slow process. Searching around I see that this error might be related [https://github.com/elastic/elasticsearch/issues/7926](https://github.com/elastic/elasticsearch/issues/7926) but it seems like that was fixed already. Any help on tracking down this issue is greatly appreciated. ---
**Author:** ![jdoss](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jdoss/32/8611_2.png) [@jdoss](https://discuss.elastic.co/u/jdoss)\ **Post date:** [April 7, 2016, 1:56pm UTC](https://discuss.elastic.co/t/reindexing-with-logstash-causes-arrayindexoutofboundsexception-errors-on-elasticsearch-1-7-5/46574/2 "2016-04-07T13:56:28Z")
Just an update on this issue. I found a workaround by creating an alias for days 1 - 15 and I can reindex off of that without seeing the ArrayIndexOutOfBoundsException errors. It is pretty slow for some reason and doing this in 15 day batches is less than optimal. ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [April 7, 2016, 3:26pm UTC](https://discuss.elastic.co/t/reindexing-with-logstash-causes-arrayindexoutofboundsexception-errors-on-elasticsearch-1-7-5/46574/3 "2016-04-07T15:26:02Z")
Could you check log files on the elasticsearch side to see if you can find the corresponding error message and post it here? ---
**Author:** ![jdoss](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jdoss/32/8611_2.png) [@jdoss](https://discuss.elastic.co/u/jdoss)\ **Post date:** [April 7, 2016, 4:21pm UTC](https://discuss.elastic.co/t/reindexing-with-logstash-causes-arrayindexoutofboundsexception-errors-on-elasticsearch-1-7-5/46574/4 "2016-04-07T16:21:56Z")
Unfortunately there is nothing on the elasticsearch side that shows the ArrayIndexOutOfBoundsException error in any of the logs. ---
**Author:** ![jdoss](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jdoss/32/8611_2.png) [@jdoss](https://discuss.elastic.co/u/jdoss)\ **Post date:** [April 7, 2016, 9:59pm UTC](https://discuss.elastic.co/t/reindexing-with-logstash-causes-arrayindexoutofboundsexception-errors-on-elasticsearch-1-7-5/46574/5 "2016-04-07T21:59:00Z")
I turned on TRACE logging on my ES cluster and also set --debug on my logstash process. Maybe this is more of an issue with Logstash since nothing is showing up on the Elasticsearch end? Here the error from logstash with debug enabled: ``` A plugin had an unrecoverable error. Will restart this plugin. Plugin: ["client-02"], index=>"logstash-2016.03.*", size=>500, scroll=>"5m", docinfo=>true, scan=>true, codec=>< LogStash::Codecs::JSON charset=>"UTF-8">, query=>"{\"query\": { \"match_all\": {} } }", docinfo_target=>"@metadata", docinfo_fields=>["_index", "_type", "_id"], ssl=>false> Error: [500] {"error":"ArrayIndexOutOfBoundsException[null]","status":500} Exception: Elasticsearch::Transport::Transport::Errors::InternalServerError Stack: /home/user/logstash-2.2.2/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/base.rb:146:in `__ raise_transport_error' /home/user/logstash-2.2.2/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/base.rb:256:in `perform_req uest' /home/user/logstash-2.2.2/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perf orm_request' /home/user/logstash-2.2.2/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/client.rb:125:in `perform_request' /home/user/logstash-2.2.2/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.0.15/lib/elasticsearch/api/actions/scroll.rb:56:in `scroll' /home/user/logstash-2.2.2/vendor/bundle/jruby/1.9/gems/logstash-input-elasticsearch-2.0.3/lib/logstash/inputs/elasticsearch.rb:196:in `scroll_request' /home/user/logstash-2.2.2/vendor/bundle/jruby/1.9/gems/logstash-input-elasticsearch-2.0.3/lib/logstash/inputs/elasticsearch.rb:165:in `process_next_scr oll' /home/user/logstash-2.2.2/vendor/bundle/jruby/1.9/gems/logstash-input-elasticsearch-2.0.3/lib/logstash/inputs/elasticsearch.rb:149:in `run' /home/user/logstash-2.2.2/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.2-java/lib/logstash/pipeline.rb:331:in `inputworker' /home/user/logstash-2.2.2/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.2-java/lib/logstash/pipeline.rb:325:in `start_input' {:level=>:error, :file=>$logstash/pipeline.rb", :line=>"342", :method=>"inputworker"} ``` The gist below contains the only thing that showed up on my ES client was after I ctrl+C'ed my logstash process. This was on the client ES node I am using as the input. I don't think it means much. > ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [April 8, 2016, 12:12am UTC](https://discuss.elastic.co/t/reindexing-with-logstash-causes-arrayindexoutofboundsexception-errors-on-elasticsearch-1-7-5/46574/6 "2016-04-08T00:12:31Z")
Did you check log files on all elasticsearch nodes in your cluster? ---
**Author:** ![jdoss](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jdoss/32/8611_2.png) [@jdoss](https://discuss.elastic.co/u/jdoss)\ **Post date:** [April 8, 2016, 1:05am UTC](https://discuss.elastic.co/t/reindexing-with-logstash-causes-arrayindexoutofboundsexception-errors-on-elasticsearch-1-7-5/46574/7 "2016-04-08T01:05:33Z")
Yep! I was tailing all of the log files on each node in the cluster. Nothing was present in the log files when Logstash kicked out the ArrayIndexOutOfBoundsException errors above. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:01pm UTC](https://discuss.elastic.co/t/reindexing-with-logstash-causes-arrayindexoutofboundsexception-errors-on-elasticsearch-1-7-5/46574/8 "2017-07-05T23:01:21Z")