Load balancing using logstash

[kriti_dabas]

I want to use two logstash so that if one goes down I can use the other one.
Is there a chance of duplicacy if i am using two logstash with same pipelines?
I will configure elasticsearch in the output of logstash.

[Badger]

@kriti_dabas

Is there a chance of duplicacy if i am using two logstash with same pipelines?

Yes. logstash does not have built-in load balancing. If you configure two logstash instances with file inputs then both will process every event. If you use something like kafka in front of logstash then that can do the load balancing for you.

[kriti_dabas]

As I said I am using two logstash from one i am able to create index but from the other one I am not.
First logstash is using ilm policy and the logs are coming second logstash simple index creation is not happening.
I am not able to create index using these configurations.
I am using the latest version 8.12.0 for both logstash and elasticsearch.
Input is connecting with kafka but output is not working.

input {
  kafka {
    bootstrap_servers => "1.1.1.141:9092,1.1.1.142:9092,1.1.1.143:9092"
    topics => ["testing"]
    decorate_events => true
    enable_auto_commit => true
    auto_offset_reset => "earliest"
    group_id => "testing_user"
    consumer_threads => 3
    client_id => logstash2
  }
}

output {
if [@metadata][kafka][topic] == "testing"{
       elasticsearch {
          hosts => ["https://1.1.1.151:9200","https://1.1.1.152:9200"]
          ssl_certificate_authorities => '/etc/logstash/certs/http_ca.crt'
          user => "elastic"
          password => "elastic"
          manage_template => false
          data_stream => false
          index => "log-testing"
         }
     }
}

Edit the yml file using vim /etc/logstash/logstash.yml:

path.data: /var/lib/logstash
pipeline.ordered: auto
pipeline.ecs_compatibility: disabled   
config.test_and_exit: false
config.reload.automatic: true
queue.type: persisted
path.queue: /var/lib/logstash/data_queue
log.level: debug
path.logs: /var/log/logstash