# Migrating data from Elastic Search 2.X to 8.X using logstash **URL:** \ **Category:** Logstash\ **Created:** [October 10, 2023, 6:10pm UTC](https://discuss.elastic.co/t/migrating-data-from-elastic-search-2-x-to-8-x-using-logstash/344760 "2023-10-10T18:10:07Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![gslp456](https://avatars.discourse-cdn.com/v4/letter/g/4da419/32.png) [@gslp456](https://discuss.elastic.co/u/gslp456)\ **Post date:** [October 10, 2023, 6:10pm UTC](https://discuss.elastic.co/t/migrating-data-from-elastic-search-2-x-to-8-x-using-logstash/344760/1 "2023-10-10T18:10:07Z")
I have an on-premise elastic cluster running on version 2.3.3. (aka old) I have another on-premise elastic cluster running on version 8.9.0 (aka new) I want to migrate data from one index of old to new using logstash 8.9.0. Below are the steps I have followed. 1. Created a piepline config file test\_migration.conf with the following fields ```auto input { elasticsearch { hosts => ["old_host"] user => "888888" password => "888888" index => "mailbox_alerts" size => 1000 scroll => "1m" } } output { elasticsearch { hosts => ["https://new_host"] cacert => '/tmp/cacert.pem' user => "888888" password => "888888" index => "test-mailbox-alerts" } } ``` 1. Ran the below command to start logstash and invoke the pipeline ```auto /els/logstash/bin/logstash --path.settings /els/Clusters/logstash -f /els/Clusters/logstash/test_migration.conf ``` This is failing due to the below error ```auto [2023-10-10T13:43:48,518][ERROR][logstash.javapipeline][main] Pipeline error {:pipeline_id=>"main", :exception=>#, :backtrace=>["/els/logstash-8.9.0/vendor/bundle/jruby/2.6.0/gems/logstash-input-elasticsearch-4.17.2/lib/logstash/inputs/elasticsearch.rb:668:in `test_connection!'", "/els/logstash-8.9.0/vendor/bundle/jruby/2.6.0/gems/logstash-input-elasticsearch-4.17.2/lib/logstash/inputs/elasticsearch.rb:314:in `register'", "/els/logstash-8.9.0/vendor/bundle/jruby/2.6.0/gems/logstash-mixin-ecs_compatibility_support-1.3.0-java/lib/logstash/plugin_mixins/ecs_compatibility_support/target_check.rb:48:in `register'", "/els/logstash-8.9.0/logstash-core/lib/logstash/java_pipeline.rb:237:in `block in register_plugins'", "org/jruby/RubyArray.java:1865:in `each'", "/els/logstash-8.9.0/logstash-core/lib/logstash/java_pipeline.rb:236:in `register_plugins'", "/els/logstash-8.9.0/logstash-core/lib/logstash/java_pipeline.rb:395:in `start_inputs'", "/els/logstash-8.9.0/logstash-core/lib/logstash/java_pipeline.rb:320:in `start_workers'", "/els/logstash-8.9.0/logstash-core/lib/logstash/java_pipeline.rb:194:in `run'", "/els/logstash-8.9.0/logstash-core/lib/logstash/java_pipeline.rb:146:in `block in start'"], "pipeline.sources"=>["/els/Clusters/logstash/test_migration.conf"], :thread=>"#"} [2023-10-10T13:43:48,519][INFO][logstash.javapipeline][main] Pipeline terminated {"pipeline.id"=>"main"} [2023-10-10T13:43:48,529][ERROR][logstash.agent] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create
, action_result: false", :backtrace=>nil} ``` What am I doing wrong here ? Does logstash 8.9.0 support migration from old to new ? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [October 10, 2023, 6:41pm UTC](https://discuss.elastic.co/t/migrating-data-from-elastic-search-2-x-to-8-x-using-logstash/344760/2 "2023-10-10T18:41:50Z")
Logstash 8.X is not compatible with Elasticsearch 2.X. There is no Logstash version that would be compatible with both Elasticsearch 2.X and 8.X. You will probably need to find other way to do this migration. The highest logstash version you can use that it is compatible with Elasticsearch 2.3 is version 5.6, according to the [support matrix](https://www.elastic.co/support/matrix#matrix_os). You will probably need to use this Logstash to consume the logs from your old Elasticsearch cluster, use a file output, and then use a newer Logstash reading those files and sending them to Elasticsearch. ---
**Author:** ![gslp456](https://avatars.discourse-cdn.com/v4/letter/g/4da419/32.png) [@gslp456](https://discuss.elastic.co/u/gslp456)\ **Post date:** [October 11, 2023, 11:01am UTC](https://discuss.elastic.co/t/migrating-data-from-elastic-search-2-x-to-8-x-using-logstash/344760/3 "2023-10-11T11:01:59Z")
Thanks for the response. I have used logstash 5.6.x to export the data of an index from elastic 2.3 to a .txt file. The size of the file is 340mb Using the above extracted file as the input , I created a pipeline configuration to copy the data to an index in Elasticsearch 8.9 cluster by using the logstash 8.9. See below. ```auto input { file { path => "/elslogs/mailbox_alerts_2.3/mailbox_alerts_23.txt" } } output { elasticsearch { hosts => ["new_host"] ssl => true cacert => '/tmp/cacert.pem' user => "888888" password => "888888" index => "test-mailbox-alerts" } stdout{} } ``` I have initialized logstash using the following command ```auto /els/logstash/bin/logstash --path.settings /els/Clusters/logstash -f /els/Clusters/logstash/test_migration.conf ``` But , I do not see the data getting imported to new 8.9 cluster. The following lines getting repeated in the logstash ```auto [2023-10-11T06:53:51,910][TRACE][filewatch.discoverer][main][1e8af91c1ecde4c83a260a6bfa969d96bb5319ef2901f0f248e4ee98e877f9b1] handling: {:new_discovery=>false, :watched_file=>""} [2023-10-11T06:54:37,996][TRACE][filewatch.tailmode.processor][main][1e8af91c1ecde4c83a260a6bfa969d96bb5319ef2901f0f248e4ee98e877f9b1] process_closed [2023-10-11T06:54:37,996][TRACE][filewatch.tailmode.processor][main][1e8af91c1ecde4c83a260a6bfa969d96bb5319ef2901f0f248e4ee98e877f9b1] process_ignored [2023-10-11T06:54:37,996][TRACE][filewatch.tailmode.processor][main][1e8af91c1ecde4c83a260a6bfa969d96bb5319ef2901f0f248e4ee98e877f9b1] process_delayed_delete [2023-10-11T06:54:37,997][TRACE][filewatch.tailmode.processor][main][1e8af91c1ecde4c83a260a6bfa969d96bb5319ef2901f0f248e4ee98e877f9b1] process_restat_for_watched_and_active [2023-10-11T06:54:37,997][TRACE][filewatch.tailmode.processor][main][1e8af91c1ecde4c83a260a6bfa969d96bb5319ef2901f0f248e4ee98e877f9b1] process_rotation_in_progress [2023-10-11T06:54:37,997][TRACE][filewatch.tailmode.processor][main][1e8af91c1ecde4c83a260a6bfa969d96bb5319ef2901f0f248e4ee98e877f9b1] process_watched [2023-10-11T06:54:37,997][TRACE][filewatch.tailmode.processor][main][1e8af91c1ecde4c83a260a6bfa969d96bb5319ef2901f0f248e4ee98e877f9b1] process_active [2023-10-11T06:54:38,997][TRACE][filewatch.tailmode.processor][main][1e8af91c1ecde4c83a260a6bfa969d96bb5319ef2901f0f248e4ee98e877f9b1] process_closed [2023-10-11T06:54:38,997][TRACE][filewatch.tailmode.processor][main][1e8af91c1ecde4c83a260a6bfa969d96bb5319ef2901f0f248e4ee98e877f9b1] process_ignored [2023-10-11T06:54:38,998][TRACE][filewatch.tailmode.processor][main][1e8af91c1ecde4c83a260a6bfa969d96bb5319ef2901f0f248e4ee98e877f9b1] process_delayed_delete [2023-10-11T06:54:38,998][TRACE][filewatch.tailmode.processor][main][1e8af91c1ecde4c83a260a6bfa969d96bb5319ef2901f0f248e4ee98e877f9b1] process_restat_for_watched_and_active [2023-10-11T06:54:38,998][TRACE][filewatch.tailmode.processor][main][1e8af91c1ecde4c83a260a6bfa969d96bb5319ef2901f0f248e4ee98e877f9b1] process_rotation_in_progress [2023-10-11T06:54:38,998][TRACE][filewatch.tailmode.processor][main][1e8af91c1ecde4c83a260a6bfa969d96bb5319ef2901f0f248e4ee98e877f9b1] process_watched [2023-10-11T06:54:38,998][TRACE][filewatch.tailmode.processor][main][1e8af91c1ecde4c83a260a6bfa969d96bb5319ef2901f0f248e4ee98e877f9b1] process_active [2023-10-11T06:54:39,788][DEBUG][org.logstash.execution.PeriodicFlush][main] Pushing flush onto pipeline. [2023-10-11T06:54:39,905][TRACE][org.logstash.instrument.metrics.ExtendedFlowMetric] RetentionWindow{policy=current id=1475958524} post-append result (captures: `8` span: `PT35.029070394S` } [2023-10-11T06:54:39,905][TRACE][org.logstash.instrument.metrics.ExtendedFlowMetric] RetentionWindow{policy=last_1_minute id=930441286} post-append result (captures: `14` span: `PT1M5.057314155S` } [2023-10-11T06:54:39,905][TRACE][org.logstash.instrument.metrics.ExtendedFlowMetric] RetentionWindow{policy=last_5_minutes id=2091489210} post-append result (captures: `35` span: `PT5M35.281983261S` } [2023-10-11T06:54:39,905][TRACE][org.logstash.instrument.metrics.ExtendedFlowMetric] RetentionWindow{policy=last_15_minutes id=1419374504} post-append result (captures: `48` span: `PT19M16.057974014S` } [2023-10-11T06:54:39,905][DEBUG][org.logstash.instrument.metrics.ExtendedFlowMetric] RetentionWindow{policy=last_15_minutes id=1419374504} forced-compaction result (captures: `38` span: `PT15M5.815007439S`) [2023-10-11T06:54:39,905][TRACE][org.logstash.instrument.metrics.ExtendedFlowMetric] RetentionWindow{policy=last_1_hour id=1405127373} post-append result (captures: `32` span: `PT27M36.722290827S` } ``` Nothing was copied to new cluster index. Any help is appreciated much ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 11, 2023, 3:48pm UTC](https://discuss.elastic.co/t/migrating-data-from-elastic-search-2-x-to-8-x-using-logstash/344760/4 "2023-10-11T15:48:38Z")
> [@gslp456](#): > > [2023-10-11T06:53:51,910][TRACE][filewatch.discoverer][main][1e8af91c1ecde4c83a260a6bfa969d96bb5319ef2901f0f248e4ee98e877f9b1] handling: {:new\_discovery=\>false, :watched\_file=\>"\"} The file input found the file and is starting to read it from 344 MB onwards. The default [start\_position](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-start_position) for a file input is end, so that it only reads data that is appended after logstash starts. If you want to read a file from the beginning then add `start_position => "beginning"` to your file input. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 8, 2023, 3:49pm UTC](https://discuss.elastic.co/t/migrating-data-from-elastic-search-2-x-to-8-x-using-logstash/344760/5 "2023-11-08T15:49:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.