Updating two different version of elastic search from one log stash

Elastic Stack Elasticsearch
1

Hi,

i have two different version of Elasticsearch, one is old(6.3.4) and another one as new(8.3.2), currently we are writting into old cluster from logstash, and are willing to miggrate to new cluster, to avoid any data loss we were looking for an approch using which we don't have any data loss, and we can just simply add an additional Elasticsearch configuration in app.conf file, same config is been shared below


input {
    file {
        start_position => "beginning"
        path => "C:/Softwares/elk-demo/elk-demo/logs/spring.log"
        sincedb_path => "NUL"
    }
}

filter {
    json {
        source => "message"
    }
}

output {
    elasticsearch {
        hosts => [http://localhost:9201]
        index => "application_logs_1"
    }
              elasticsearch {
        hosts => [http://localhost:9202]
        index => "application_logs_2"
    }
    stdout {}
}

Question 1st:- can we write into two Elasticsearch cluster from one logstash, be updating and passing the same in config file of logstash?

2

Unfortunately you cannot do this from a single instance of Logstash due to compatibility issues.

You might need to send the data to a second Logstash instance that handles the newer version of Elasticsearch.

3

As Warkolm already said, it is not possible with those versions.

The earliest version of Logstash that is compatible with Elasticsearch 8.X is 7.17.0, but Logstash 7.17.0 is not compatible with Elasticsearch 6.X, the latest version that is compatible is Logstash 6.8.X.

One easy way to solve your issue is to add Kafka in your infrastructure.

You would a Logstash 6.8.X to read your file in send it to a Kafka topic, then in the same Logstash you could have a pipeline that would read from this topic and send the data to your Elasticsearch 6.3.4

You would then spin-up another Logstash instance, now on version 8.3.2 (or 8.5.1, which is the latest) to read from the same Kafka Topic and send the data to the new cluster.