Migrating data from Elastic Search 2.X to 8.X using logstash

Elastic Stack Logstash
1

I have an on-premise elastic cluster running on version 2.3.3. (aka old)
I have another on-premise elastic cluster running on version 8.9.0 (aka new)
I want to migrate data from one index of old to new using logstash 8.9.0. Below are the steps I have followed.

  1. Created a piepline config file test_migration.conf with the following fields
input {
    elasticsearch {
        hosts => ["old_host"]
        user => "888888"
        password => "888888"
        index => "mailbox_alerts"
        size => 1000
        scroll => "1m"
    }
}
output {
    elasticsearch {
        hosts => ["https://new_host"]
        cacert => '/tmp/cacert.pem'
        user => "888888"
        password => "888888"
        index => "test-mailbox-alerts"
    }
}
  1. Ran the below command to start logstash and invoke the pipeline
/els/logstash/bin/logstash --path.settings /els/Clusters/logstash -f /els/Clusters/logstash/test_migration.conf

This is failing due to the below error

[2023-10-10T13:43:48,518][ERROR][logstash.javapipeline    ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<LogStash::ConfigurationError: Could not connect to a compatible version of Elasticsearch>, :backtrace=>["/els/logstash-8.9.0/vendor/bundle/jruby/2.6.0/gems/logstash-input-elasticsearch-4.17.2/lib/logstash/inputs/elasticsearch.rb:668:in `test_connection!'", "/els/logstash-8.9.0/vendor/bundle/jruby/2.6.0/gems/logstash-input-elasticsearch-4.17.2/lib/logstash/inputs/elasticsearch.rb:314:in `register'", "/els/logstash-8.9.0/vendor/bundle/jruby/2.6.0/gems/logstash-mixin-ecs_compatibility_support-1.3.0-java/lib/logstash/plugin_mixins/ecs_compatibility_support/target_check.rb:48:in `register'", "/els/logstash-8.9.0/logstash-core/lib/logstash/java_pipeline.rb:237:in `block in register_plugins'", "org/jruby/RubyArray.java:1865:in `each'", "/els/logstash-8.9.0/logstash-core/lib/logstash/java_pipeline.rb:236:in `register_plugins'", "/els/logstash-8.9.0/logstash-core/lib/logstash/java_pipeline.rb:395:in `start_inputs'", "/els/logstash-8.9.0/logstash-core/lib/logstash/java_pipeline.rb:320:in `start_workers'", "/els/logstash-8.9.0/logstash-core/lib/logstash/java_pipeline.rb:194:in `run'", "/els/logstash-8.9.0/logstash-core/lib/logstash/java_pipeline.rb:146:in `block in start'"], "pipeline.sources"=>["/els/Clusters/logstash/test_migration.conf"], :thread=>"#<Thread:0x7452137f@/els/logstash-8.9.0/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[2023-10-10T13:43:48,519][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2023-10-10T13:43:48,529][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}

What am I doing wrong here ? Does logstash 8.9.0 support migration from old to new ?

2

Logstash 8.X is not compatible with Elasticsearch 2.X.

There is no Logstash version that would be compatible with both Elasticsearch 2.X and 8.X.

You will probably need to find other way to do this migration.

The highest logstash version you can use that it is compatible with Elasticsearch 2.3 is version 5.6, according to the support matrix.

You will probably need to use this Logstash to consume the logs from your old Elasticsearch cluster, use a file output, and then use a newer Logstash reading those files and sending them to Elasticsearch.

3

Thanks for the response.
I have used logstash 5.6.x to export the data of an index from elastic 2.3 to a .txt file. The size of the file is 340mb

Using the above extracted file as the input , I created a pipeline configuration to copy the data to an index in Elasticsearch 8.9 cluster by using the logstash 8.9. See below.

input {
    file {
        path => "/elslogs/mailbox_alerts_2.3/mailbox_alerts_23.txt"
    }
}

output {
    elasticsearch {
        hosts => ["new_host"]
        ssl => true
        cacert => '/tmp/cacert.pem'
        user => "888888"
        password => "888888"
        index => "test-mailbox-alerts"
    }
    stdout{}
}

I have initialized logstash using the following command

/els/logstash/bin/logstash --path.settings /els/Clusters/logstash -f /els/Clusters/logstash/test_migration.conf

But , I do not see the data getting imported to new 8.9 cluster. The following lines getting repeated in the logstash

[2023-10-11T06:53:51,910][TRACE][filewatch.discoverer     ][main][1e8af91c1ecde4c83a260a6bfa969d96bb5319ef2901f0f248e4ee98e877f9b1] handling: {:new_discovery=>false, :watched_file=>"<FileWatch::WatchedFile: @filename='mailbox_alerts_23.txt', @state=:ignored, @recent_states=[:watched, :watched], @bytes_read=360479689, @bytes_unread=0, current_size=360479689, last_stat_size=360479689, file_open?=false, @initial=false, sincedb_key='16784284 0 64772'>"}

[2023-10-11T06:54:37,996][TRACE][filewatch.tailmode.processor][main][1e8af91c1ecde4c83a260a6bfa969d96bb5319ef2901f0f248e4ee98e877f9b1] process_closed
[2023-10-11T06:54:37,996][TRACE][filewatch.tailmode.processor][main][1e8af91c1ecde4c83a260a6bfa969d96bb5319ef2901f0f248e4ee98e877f9b1] process_ignored
[2023-10-11T06:54:37,996][TRACE][filewatch.tailmode.processor][main][1e8af91c1ecde4c83a260a6bfa969d96bb5319ef2901f0f248e4ee98e877f9b1] process_delayed_delete
[2023-10-11T06:54:37,997][TRACE][filewatch.tailmode.processor][main][1e8af91c1ecde4c83a260a6bfa969d96bb5319ef2901f0f248e4ee98e877f9b1] process_restat_for_watched_and_active
[2023-10-11T06:54:37,997][TRACE][filewatch.tailmode.processor][main][1e8af91c1ecde4c83a260a6bfa969d96bb5319ef2901f0f248e4ee98e877f9b1] process_rotation_in_progress
[2023-10-11T06:54:37,997][TRACE][filewatch.tailmode.processor][main][1e8af91c1ecde4c83a260a6bfa969d96bb5319ef2901f0f248e4ee98e877f9b1] process_watched
[2023-10-11T06:54:37,997][TRACE][filewatch.tailmode.processor][main][1e8af91c1ecde4c83a260a6bfa969d96bb5319ef2901f0f248e4ee98e877f9b1] process_active
[2023-10-11T06:54:38,997][TRACE][filewatch.tailmode.processor][main][1e8af91c1ecde4c83a260a6bfa969d96bb5319ef2901f0f248e4ee98e877f9b1] process_closed
[2023-10-11T06:54:38,997][TRACE][filewatch.tailmode.processor][main][1e8af91c1ecde4c83a260a6bfa969d96bb5319ef2901f0f248e4ee98e877f9b1] process_ignored
[2023-10-11T06:54:38,998][TRACE][filewatch.tailmode.processor][main][1e8af91c1ecde4c83a260a6bfa969d96bb5319ef2901f0f248e4ee98e877f9b1] process_delayed_delete
[2023-10-11T06:54:38,998][TRACE][filewatch.tailmode.processor][main][1e8af91c1ecde4c83a260a6bfa969d96bb5319ef2901f0f248e4ee98e877f9b1] process_restat_for_watched_and_active
[2023-10-11T06:54:38,998][TRACE][filewatch.tailmode.processor][main][1e8af91c1ecde4c83a260a6bfa969d96bb5319ef2901f0f248e4ee98e877f9b1] process_rotation_in_progress
[2023-10-11T06:54:38,998][TRACE][filewatch.tailmode.processor][main][1e8af91c1ecde4c83a260a6bfa969d96bb5319ef2901f0f248e4ee98e877f9b1] process_watched
[2023-10-11T06:54:38,998][TRACE][filewatch.tailmode.processor][main][1e8af91c1ecde4c83a260a6bfa969d96bb5319ef2901f0f248e4ee98e877f9b1] process_active
[2023-10-11T06:54:39,788][DEBUG][org.logstash.execution.PeriodicFlush][main] Pushing flush onto pipeline.
[2023-10-11T06:54:39,905][TRACE][org.logstash.instrument.metrics.ExtendedFlowMetric] RetentionWindow{policy=current id=1475958524} post-append result (captures: `8` span: `PT35.029070394S` }
[2023-10-11T06:54:39,905][TRACE][org.logstash.instrument.metrics.ExtendedFlowMetric] RetentionWindow{policy=last_1_minute id=930441286} post-append result (captures: `14` span: `PT1M5.057314155S` }
[2023-10-11T06:54:39,905][TRACE][org.logstash.instrument.metrics.ExtendedFlowMetric] RetentionWindow{policy=last_5_minutes id=2091489210} post-append result (captures: `35` span: `PT5M35.281983261S` }
[2023-10-11T06:54:39,905][TRACE][org.logstash.instrument.metrics.ExtendedFlowMetric] RetentionWindow{policy=last_15_minutes id=1419374504} post-append result (captures: `48` span: `PT19M16.057974014S` }
[2023-10-11T06:54:39,905][DEBUG][org.logstash.instrument.metrics.ExtendedFlowMetric] RetentionWindow{policy=last_15_minutes id=1419374504} forced-compaction result (captures: `38` span: `PT15M5.815007439S`)
[2023-10-11T06:54:39,905][TRACE][org.logstash.instrument.metrics.ExtendedFlowMetric] RetentionWindow{policy=last_1_hour id=1405127373} post-append result (captures: `32` span: `PT27M36.722290827S` }

Nothing was copied to new cluster index. Any help is appreciated much

4

The file input found the file and is starting to read it from 344 MB onwards. The default start_position for a file input is end, so that it only reads data that is appended after logstash starts. If you want to read a file from the beginning then add start_position => "beginning" to your file input.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.