# Configure 2 log paths for filebeat and output 2 kafka topics and 2 indexes to display on kibana

**URL:** <https://discuss.elastic.co/t/configure-2-log-paths-for-filebeat-and-output-2-kafka-topics-and-2-indexes-to-display-on-kibana/346941>\
**Category:** Beats\
**Tags:** elastic-stack-monitoring, elastic-stack-alerting, filebeat\
**Created:** [November 13, 2023, 4:16am UTC](https://discuss.elastic.co/t/configure-2-log-paths-for-filebeat-and-output-2-kafka-topics-and-2-indexes-to-display-on-kibana/346941 "2023-11-13T04:16:18Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nghia\_D\_ng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nghia_d_ng/32/123483_2.png) [@Nghia\_D\_ng](https://discuss.elastic.co/u/Nghia_D_ng)\
**Post date:** [November 13, 2023, 4:16am UTC](https://discuss.elastic.co/t/configure-2-log-paths-for-filebeat-and-output-2-kafka-topics-and-2-indexes-to-display-on-kibana/346941/1 "2023-11-13T04:16:18Z")

</div>

Hi team Elastic,

I am configuring ELK according to the following flow:  
Filebeat\> kafka\>Logstash\>ES\>Kibana  
I need to monitor 2 logs paths on the server on kibana  
I want to configure those 2 paths as 2 separate kafka topics, output elasticsearch 2 separate indexes.

filebeat.yml

```auto
filebeat.inputs:
- type: log
  paths:
    - /a/b/c_d_01*
  fields:
    log_type: c_d_01
  fields_under_root: true  
output.kafka:
    hosts: ["IP:9092"]
    topic: "app01"

filebeat.inputs:
- type: log
  paths:
    - /a/b/c_d_02*
  fields:
    log_type: c_d_02
  fields_under_root: true  
output.kafka:
    hosts: ["IP:9092"]
    topic: "app02"

```

logstash.conf

```auto
input {
    kafka {
            bootstrap_servers => 'IP:9092'
            topics => ["app01", "app02"]
            codec => json {}
          }
}

filter {
  # Add any necessary filters here
}

output {
  if [kafka][topic] == "app01" {
    elasticsearch {
      hosts => ["IP:9200"]
      sniffing => true
      index => "app01-%{+YYYY.MM.dd}"
      user => "elastic"
      password => "abcd@123"
      ssl => true
      ssl_certificate_verification => false
      cacert => '/u01/logstash/certssl/elasticsearch-ca.pem'
    }
  } else if [kafka][topic] == "app02" {
    elasticsearch {
      hosts => ["IP:9200"]
      sniffing => true
      index => "app02-%{+YYYY.MM.dd}"
      user => "elastic"
      password => "abcd@123"
      ssl => true
      ssl_certificate_verification => false
      cacert => '/u01/logstash/certssl/elasticsearch-ca.pem'
    }
  }
}

```

Please help me!

---

<div class="post-metadata">

**Author:** ![Nghia\_D\_ng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nghia_d_ng/32/123483_2.png) [@Nghia\_D\_ng](https://discuss.elastic.co/u/Nghia_D_ng)\
**Post date:** [November 13, 2023, 4:29am UTC](https://discuss.elastic.co/t/configure-2-log-paths-for-filebeat-and-output-2-kafka-topics-and-2-indexes-to-display-on-kibana/346941/2 "2023-11-13T04:29:24Z")

</div>

Currently I'm doing it this way  
But with only 1 index, I want to separate 2 indexes to facilitate checking logs on kibana

filebeat.yml

```auto
filebeat.inputs:
- type: log
  paths:
    - /a/b/c_d_01*
- type: log
  paths:
    - /a/b/c_d_07*

output.kafka:
  hosts: ["IP:9092"]
  topic: app01-app02

```

logstash.conf

```auto
input {
    kafka {
            bootstrap_servers => 'IP:9092'
            topics => ["app01-app02"]
            codec => json {}
            group_id => ["app01-app02"]
            auto_offset_reset => earliest
          }
}

output {
    elasticsearch {
      hosts => ["IP:9200"]
      sniffing => true
      index => "app01-app02-%{+YYYY.MM.dd}"
      user => "elastic"
      password => "abcd@123"
      ssl => true
      ssl_certificate_verification => false
      cacert => '/u01/logstash/certssl/elasticsearch-ca.pem'
    }

```

---

<div class="post-metadata">

**Author:** ![Nghia\_D\_ng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nghia_d_ng/32/123483_2.png) [@Nghia\_D\_ng](https://discuss.elastic.co/u/Nghia_D_ng)\
**Post date:** [November 13, 2023, 8:40am UTC](https://discuss.elastic.co/t/configure-2-log-paths-for-filebeat-and-output-2-kafka-topics-and-2-indexes-to-display-on-kibana/346941/3 "2023-11-13T08:40:23Z")

</div>

Hi @carly.richmond , @tsullivan , @stephenb ,  
Please help me with this problem  
Or does anyone in your team know it?

---

<div class="post-metadata">

**Author:** ![Nghia\_D\_ng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nghia_d_ng/32/123483_2.png) [@Nghia\_D\_ng](https://discuss.elastic.co/u/Nghia_D_ng)\
**Post date:** [November 13, 2023, 8:45am UTC](https://discuss.elastic.co/t/configure-2-log-paths-for-filebeat-and-output-2-kafka-topics-and-2-indexes-to-display-on-kibana/346941/4 "2023-11-13T08:45:32Z")

</div>

Sorry for the rush, but I really need someone's help with this. Thank you everyone

---

<div class="post-metadata">

**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\
**Post date:** [November 13, 2023, 10:16am UTC](https://discuss.elastic.co/t/configure-2-log-paths-for-filebeat-and-output-2-kafka-topics-and-2-indexes-to-display-on-kibana/346941/5 "2023-11-13T10:16:04Z")

</div>

Hi @Nghia_D_ng,

Thanks for your query. This forum is best effort help, and not intended as support. It can take some time to get an answer, so please don't ping team members directly unless it's been outstanding for a couple of days.

To confirm, you want to have logs from each kafka topic go to separate Elasticsearch indexes? You should be able to do that using conditional output similar to what you had in your first post.

Was there a particular issue you had with that configuration?

---

<div class="post-metadata">

**Author:** ![Nghia\_D\_ng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nghia_d_ng/32/123483_2.png) [@Nghia\_D\_ng](https://discuss.elastic.co/u/Nghia_D_ng)\
**Post date:** [November 13, 2023, 11:57am UTC](https://discuss.elastic.co/t/configure-2-log-paths-for-filebeat-and-output-2-kafka-topics-and-2-indexes-to-display-on-kibana/346941/6 "2023-11-13T11:57:34Z")

</div>

HI @carly.richmond ,  
When configured like the filebeat.yml file I mentioned above, the status fails. I think it's error somewhere

---

<div class="post-metadata">

**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\
**Post date:** [November 14, 2023, 10:54am UTC](https://discuss.elastic.co/t/configure-2-log-paths-for-filebeat-and-output-2-kafka-topics-and-2-indexes-to-display-on-kibana/346941/7 "2023-11-14T10:54:20Z")

</div>

Ok, are there any errors in the logs? Can you share the error?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [November 14, 2023, 1:42pm UTC](https://discuss.elastic.co/t/configure-2-log-paths-for-filebeat-and-output-2-kafka-topics-and-2-indexes-to-display-on-kibana/346941/8 "2023-11-14T13:42:24Z")

</div>

> [@Nghia\_D\_ng](#):
>
> When configured like the filebeat.yml file I mentioned above, the status fails. I think it's error somewhere

The first `filebeat.yml` you shared will not work, you can have only one output, you configured two, this is not supported.

You can use a conditional, the [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/kafka-output.html#topics-option-kafka) has an example about how to use a conditional in the kafka output.

You may use the value that you are adding in the `log_type` to filter it.

---

<div class="post-metadata">

**Author:** ![Nghia\_D\_ng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nghia_d_ng/32/123483_2.png) [@Nghia\_D\_ng](https://discuss.elastic.co/u/Nghia_D_ng)\
**Post date:** [November 15, 2023, 7:38am UTC](https://discuss.elastic.co/t/configure-2-log-paths-for-filebeat-and-output-2-kafka-topics-and-2-indexes-to-display-on-kibana/346941/9 "2023-11-15T07:38:39Z")

</div>

Hi @leandrojmp ,  
Thank you for your support  
I succeed

---

<div class="post-metadata">

**Author:** ![Nghia\_D\_ng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nghia_d_ng/32/123483_2.png) [@Nghia\_D\_ng](https://discuss.elastic.co/u/Nghia_D_ng)\
**Post date:** [November 15, 2023, 7:38am UTC](https://discuss.elastic.co/t/configure-2-log-paths-for-filebeat-and-output-2-kafka-topics-and-2-indexes-to-display-on-kibana/346941/10 "2023-11-15T07:38:52Z")

</div>

Hi @carly.richmond ,  
Thank you for your support  
I succeed

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 13, 2023, 9:39am UTC](https://discuss.elastic.co/t/configure-2-log-paths-for-filebeat-and-output-2-kafka-topics-and-2-indexes-to-display-on-kibana/346941/11 "2023-12-13T09:39:44Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.