# Filebeat 7.12 to output to multiple kafka topics

**URL:** <https://discuss.elastic.co/t/filebeat-7-12-to-output-to-multiple-kafka-topics/288056>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [October 30, 2021, 8:30am UTC](https://discuss.elastic.co/t/filebeat-7-12-to-output-to-multiple-kafka-topics/288056 "2021-10-30T08:30:10Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![itsec](https://avatars.discourse-cdn.com/v4/letter/i/77aa72/32.png) [@itsec](https://discuss.elastic.co/u/itsec)\
**Post date:** [October 30, 2021, 8:30am UTC](https://discuss.elastic.co/t/filebeat-7-12-to-output-to-multiple-kafka-topics/288056/1 "2021-10-30T08:30:10Z")

</div>

Hi,  
I'm on filebeat 7.12 (Centos7). How do i output to multiple topics  
Pls advise.

filebeat:  
prospectors:  
- type: log  
enabled: true  
paths:  
-/opt/xssssss/log/xxx.txt  
close\_rename: true  
fields:  
document\_type: "xxxxxx1test"  
tail\_files: true  
- type: log  
enabled: true  
paths:  
-/var/log/mesg  
fields:  
document\_type: "xxxxxx1topic"  
tail\_files: true

output.kafka:  
enabled: true  
hosts: ["x.x.x.x:9092", "x.x.x.x:9092", "x.x.x.x:9092"]  
topic: "%{[type]}"  
partition.round\_robin:  
reachable\_only: false  
required\_acks: 1

log error show -\> Exiting: error loading config file: yaml: line 37: mapping values are not allowed in this context and line 37 is _enabled: true_

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [October 31, 2021, 2:52pm UTC](https://discuss.elastic.co/t/filebeat-7-12-to-output-to-multiple-kafka-topics/288056/2 "2021-10-31T14:52:57Z")

</div>

Well you did not format you code using the `</>` button so we can not tell if it is valid but the error code indicates it is not.

YAML needs exact indentation ,/ syntax pretty sure yours is not, that is what the error is saying.

Also I noticed for this line it should be single quotes per the documentation example [here](https://www.elastic.co/guide/en/beats/filebeat/current/kafka-output.html)

`topic: '%{[fields.log_topic]}'`

---

<div class="post-metadata">

**Author:** ![itsec](https://avatars.discourse-cdn.com/v4/letter/i/77aa72/32.png) [@itsec](https://discuss.elastic.co/u/itsec)\
**Post date:** [November 7, 2021, 7:38am UTC](https://discuss.elastic.co/t/filebeat-7-12-to-output-to-multiple-kafka-topics/288056/3 "2021-11-07T07:38:15Z")

</div>

My bad. i hope this helps

```auto
 filebeat:
   prospectors:
        - type: log
            enabled: true
            paths: 
                -/xx/xx/xxx.txt
            close_rename: true
            fields:
                document_type: "xxxxx1test"
            tail_files: true
        - type: log
            enabled: true
            paths:
                -/var/log/mesg
            fields:
                document_type: "xxxx1topic"   
            tail_files: true

  scan_frequency: 10s
  backoff: 1s
  close_inactive: 1m
  ignore_older: 2h
  clean_inactive: 25h

output.kafka:
  enabled: true

  hosts: ["192.xxx.xxx.120:9092", "192.xxx.xxx.122:9092", "192.xxx.xxx.123:9092"]
  topic: "%{[type]}"
  partition.round_robin:
     reachable_only: false
  required_acks: 1

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 7, 2021, 3:14pm UTC](https://discuss.elastic.co/t/filebeat-7-12-to-output-to-multiple-kafka-topics/288056/4 "2021-11-07T15:14:28Z")

</div>

Apologies I am unclear what you are asking that seems to be only a partial filebeat.yml file and it is not correctly formatted in any way.

I am not sure where you got `prospectors` from

This command can be used to test the validity of your configuration

`./filebeat test config -c ./filebeat.yml`

I suggest perhaps taking a closer look at the documentation and starting from the filebeat.yml that comes with the distribution and working .

Here is a filebeat.yml that passes parsing test perhaps it will help.

```auto
filebeat.inputs:

    - type: log
      enabled: true
      paths: 
        -/xx/xx/xxx.txt
      close_rename: true
      fields:
        document_type: "xxxxx1test"
      tail_files: true
      scan_frequency: 10s
      backoff: 1s
      close_inactive: 1m
      ignore_older: 2h
      clean_inactive: 25h

    - type: log
      enabled: true
      paths:
       -/var/log/mesg
      fields:
        document_type: "xxxx1topic"  
      tail_files: true
      scan_frequency: 10s
      backoff: 1s
      close_inactive: 1m
      ignore_older: 2h
      clean_inactive: 25h

output.kafka:
  enabled: true

  hosts: ["192.xxx.xxx.120:9092", "192.xxx.xxx.122:9092", "192.xxx.xxx.123:9092"]
  topic: "%{[fields.document_type]}"
  partition.round_robin:
    reachable_only: false
  required_acks: 1

```

---

<div class="post-metadata">

**Author:** ![itsec](https://avatars.discourse-cdn.com/v4/letter/i/77aa72/32.png) [@itsec](https://discuss.elastic.co/u/itsec)\
**Post date:** [November 8, 2021, 1:51am UTC](https://discuss.elastic.co/t/filebeat-7-12-to-output-to-multiple-kafka-topics/288056/5 "2021-11-08T01:51:39Z")

</div>

Hi Stephen, i have a working config with only single output to a kafka topic and i wish to output to 2 different topics. I picked up some examples from Internet and it claimed this config worked.

Here's the error  
Exiting: error loading config file: yaml: line 37: mapping values are not allowed in this context

and line 37 is referred "enabled: true"

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 8, 2021, 2:01am UTC](https://discuss.elastic.co/t/filebeat-7-12-to-output-to-multiple-kafka-topics/288056/6 "2021-11-08T02:01:10Z")

</div>

Show me your _entire exact_ working config `filebeat.yml` with 1 output that is working then show me your _exact_ config trying multiple.. That first config you showed would have never worked it was invalid.

> [@itsec](#):
>
> Exiting: error loading config file: yaml: line 37: mapping values are not allowed in this context

That is a simple direct yaml syntax error...

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [November 8, 2021, 3:59am UTC](https://discuss.elastic.co/t/filebeat-7-12-to-output-to-multiple-kafka-topics/288056/7 "2021-11-08T03:59:20Z")

</div>

The config you shared has only 32 lines, so you didn't share the full config or you are running another config file for some reason.

Also, `prospectors` was changed to `inputs` in version `6.3` or `6.4`, I'm not sure if the newer versions still works with the configuration using `prospectors`.

When checking examples from internet it is always good to look into the official documentation of the version you are using to see if the example still applies, the tools in Elastic Stack moves pretty fast and a lot of things can get deprecated or not work anymore.

As asked, share your working config and the config you are trying to run.

---

<div class="post-metadata">

**Author:** ![itsec](https://avatars.discourse-cdn.com/v4/letter/i/77aa72/32.png) [@itsec](https://discuss.elastic.co/u/itsec)\
**Post date:** [November 11, 2021, 2:47pm UTC](https://discuss.elastic.co/t/filebeat-7-12-to-output-to-multiple-kafka-topics/288056/8 "2021-11-11T14:47:38Z")

</div>

Hi, here's my working copy.  
The objective is output to various kafka topic based on different inputs.

```auto
filebeat.inputs:
- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
     - /xxx/xxx/xxx.txt

  scan_frequency: 10s
  backoff: 1s
  close_inactive: 1m
  ignore_older: 2h
  clean_inactive: 25h

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: true

  # Period on which files under path should be checked for changes
  reload.period: 10s

# -------------------------------- Kafka Output --------------------------------
  output.kafka:
  enabled: true

  hosts: ["192.xxx.xxx.120:9092", "192.xxx.xxx.122:9092", "192.xxx.xxx.123:9092"]
  topic: "xxxxx1test"

```

---

<div class="post-metadata">

**Author:** ![itsec](https://avatars.discourse-cdn.com/v4/letter/i/77aa72/32.png) [@itsec](https://discuss.elastic.co/u/itsec)\
**Post date:** [November 20, 2021, 8:00am UTC](https://discuss.elastic.co/t/filebeat-7-12-to-output-to-multiple-kafka-topics/288056/9 "2021-11-20T08:00:49Z")

</div>

anyone knows? i had been googling and i couldnt find except those with examaples with prospectors

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [November 20, 2021, 2:37pm UTC](https://discuss.elastic.co/t/filebeat-7-12-to-output-to-multiple-kafka-topics/288056/10 "2021-11-20T14:37:48Z")

</div>

Are you sure you are running the config you shared?

You said that you got this error:

```auto
Exiting: error loading config file: yaml: line 37: mapping values are not allowed in this context

```

The filebeat configuration you shared only have 33 lines, so it can't be this config that is giving you that error.

To output to different kafka topics according to an field is [documented](https://www.elastic.co/guide/en/beats/filebeat/master/kafka-output.html#kafka-output), the example configuration that Stephen will do that.

You need first to have a field to filter in your document, you can add a field with this:

```auto
      fields:
        kafka_topic: "topic-name"

```

Then in the Kafka output you will refer to this field this way:

```auto
output.kafka:
  hosts: ["kafka-brokers"]
  topic: '%{[fields.kafka_topic]}'

```

There's nothing else to do, you just to make sure what config you are running, you share a config with 33 lines and you are saying that you got an error at line 37, this does not match.

---

<div class="post-metadata">

**Author:** ![itsec](https://avatars.discourse-cdn.com/v4/letter/i/77aa72/32.png) [@itsec](https://discuss.elastic.co/u/itsec)\
**Post date:** [November 22, 2021, 9:42am UTC](https://discuss.elastic.co/t/filebeat-7-12-to-output-to-multiple-kafka-topics/288056/11 "2021-11-22T09:42:35Z")

</div>

Hi Leandro, its indeed line #37 and i remove default comments that comes with the installation.

---

<div class="post-metadata">

**Author:** ![itsec](https://avatars.discourse-cdn.com/v4/letter/i/77aa72/32.png) [@itsec](https://discuss.elastic.co/u/itsec)\
**Post date:** [November 22, 2021, 10:23am UTC](https://discuss.elastic.co/t/filebeat-7-12-to-output-to-multiple-kafka-topics/288056/12 "2021-11-22T10:23:15Z")

</div>

Hi Leandro, Thank you for your help. I'm able to output 2 different logs to different topic.  
Thank you 🙂

```auto
- type: log

  # Change to true to enable this input configuration.
  enabled: true

  fields:
        kafka_topic: "xxxxx1test"
  # Paths that should be crawled and fetched. Glob based paths.
  paths:
     - /xxx/xxx/xxx/xxx/xxxx.txt

  fields:
        kafka_topic: "xxxxx1topic"

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/messages

 

  scan_frequency: 10s
  backoff: 1s
  close_inactive: 1m
  ignore_older: 2h
  clean_inactive: 25h

output.kafka:
  enabled: true
  hosts: ["kafka-brokers"]
  topic: '%{[fields.kafka_topic]}'

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 20, 2021, 12:23pm UTC](https://discuss.elastic.co/t/filebeat-7-12-to-output-to-multiple-kafka-topics/288056/13 "2021-12-20T12:23:40Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.