# Modular pipeline construction with same input plugin (Kafka) but different topic name

**URL:** https://discuss.elastic.co/t/modular-pipeline-construction-with-same-input-plugin-kafka-but-different-topic-name/310300
**Category:** Logstash
**Created:** [July 21, 2022, 1:26pm UTC](https://discuss.elastic.co/t/modular-pipeline-construction-with-same-input-plugin-kafka-but-different-topic-name/310300 "2022-07-21T13:26:37Z")
**Posts on this page:** 2
**Page:** 1

<div class="post-metadata">

### Author: ![ibrahim.ramadan](https://avatars.discourse-cdn.com/v4/letter/i/91b2a8/32.png) [@ibrahim.ramadan](https://discuss.elastic.co/u/ibrahim.ramadan)
#### Post date: [July 21, 2022, 1:26pm UTC](https://discuss.elastic.co/t/modular-pipeline-construction-with-same-input-plugin-kafka-but-different-topic-name/310300/1 "2022-07-21T13:26:37Z")

</div>

Hi,

I am trying to construct modular multiple pipeline in logstash. I have the same input plugin from (kafka) in all pipelines but at each one some attributes like topic name and broker address different from each input plugin file

the following example for illustrate the problem.

this is my pipeline.yml

```auto
- pipeline.id: pipeline_1
  path.config: "/etc/logstash/conf.d/{01_in,01_filter1,02_filter2,01_out}.conf"

- pipeline.id: pipeline_2
  path.config: "/etc/logstash/conf.d/{02_in,03_filter3,02_filter2,02_out}.conf"

```

this is my input configuration files

01\_in.conf file like the following

```auto
input {
    kafka {
        bootstrap_servers => "10.89.16.15:9092"
        topics => ["topic1"]
        enable_auto_commit => "false"
        auto_offset_reset => "earliest"
        group_id => "group-1"
    }
}

```

02\_in.conf file like the following

```auto
input {
    kafka {
        bootstrap_servers => "10.89.16.12:9092"
        topics => ["topic2"]    
        enable_auto_commit => "false"
        auto_offset_reset => "earliest"
        group_id => "group-1"
    }
}

```

* * *

notice that bootstrap\_server and topics are different from each file

**my question is if I have 100 topic I will create 100 input config file which is have only 2 different fields, so i need an idea that i can passing topic name and server address to generic input.conf configuration file or pipeline.yml file or something like that**

thanks in advance

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 18, 2022, 1:26pm UTC](https://discuss.elastic.co/t/modular-pipeline-construction-with-same-input-plugin-kafka-but-different-topic-name/310300/2 "2022-08-18T13:26:44Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.