# Configure two sets of hosts for different logs **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [April 24, 2024, 6:52am UTC](https://discuss.elastic.co/t/configure-two-sets-of-hosts-for-different-logs/358094 "2024-04-24T06:52:02Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![swtg](https://avatars.discourse-cdn.com/v4/letter/s/bbce88/32.png) [@swtg](https://discuss.elastic.co/u/swtg)\ **Post date:** [April 24, 2024, 6:52am UTC](https://discuss.elastic.co/t/configure-two-sets-of-hosts-for-different-logs/358094/1 "2024-04-24T06:52:02Z")
Hi Team, I am looking to configure 2 sets of kafka hosts to which different logs will be sent. Example: filebeat.inputs: syslog this should be sent via below hosts and without any authentication (no auth configured on server end) output.kafka: topic: topic-syslog hosts: - syslog-kafka:2222 topics: - topic: topic-syslog when.contains: fileset.name: "syslog" filebeat.inputs: filestream for maillog this should be sent via different kafka hosts on which auth is configured. This option work if it solely configured. However I cant get it work it together with syslog. output.kafka: topic: topic-maillog sasl.mechanism: PLAIN username: "abc" password: "xx" hosts: - maillog-kafka:2222 topics: - topic: topic-maillog when.contains: input.id: "maillog" Can anyone please help how I can set it up together so that syslog goes via syslog-kfka and maillog go via maillog-kafka? ---
**Author:** ![Adriann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adriann/32/77780_2.png) [@Adriann](https://discuss.elastic.co/u/Adriann)\ **Post date:** [April 24, 2024, 9:45am UTC](https://discuss.elastic.co/t/configure-two-sets-of-hosts-for-different-logs/358094/2 "2024-04-24T09:45:18Z")
@swtg Please edit the post and format properly the configuration. ---
**Author:** ![Adriann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adriann/32/77780_2.png) [@Adriann](https://discuss.elastic.co/u/Adriann)\ **Post date:** [April 24, 2024, 9:47am UTC](https://discuss.elastic.co/t/configure-two-sets-of-hosts-for-different-logs/358094/3 "2024-04-24T09:47:12Z")
```auto filebeat.inputs: - type: syslog enabled: true - type: filestream enabled: true paths: - /path/to/maillog output.kafka: hosts: - syslog-kafka:2222 - maillog-kafka:2222 topic: '%{[fields.topic]}' producer: compression: gzip max_message_bytes: 1000000 ssl.enabled: true sasl.mechanism: PLAIN sasl.username: "abc" sasl.password: "xx" topics: - topic: topic-syslog when.contains: fileset.name: "syslog" sasl.enabled: false - topic: topic-maillog when.contains: input.id: "maillog" sasl.enabled: true ``` Explanation: 1. In the `filebeat.inputs` section, we define two input sources: - `type: syslog` for collecting syslog messages - `type: filestream` for collecting maillog files from a specific path 1. In the `output.kafka` section, we specify the Kafka hosts to which the logs will be sent: - `syslog-kafka:2222` for syslog messages - `maillog-kafka:2222` for maillog messages 1. We use the `topic` option with the value `'%{[fields.topic]}'` to dynamically set the topic based on the `fields.topic` field. 2. We configure the Kafka producer settings such as compression and maximum message size. 3. We enable SSL for secure communication with Kafka. 4. We set the SASL authentication mechanism to `PLAIN` and provide the username and password for authentication. 5. In the `topics` section, we define two topic configurations: - For `topic-syslog`, we use the `when.contains` condition to match the `fileset.name` field with "syslog". We set `sasl.enabled: false` to disable SASL authentication for this topic. - For `topic-maillog`, we use the `when.contains` condition to match the `input.id` field with "maillog". We set `sasl.enabled: true` to enable SASL authentication for this topic. With this configuration, Filebeat will send syslog messages to the `topic-syslog` topic on the `syslog-kafka:2222` host without authentication, and maillog messages to the `topic-maillog` topic on the `maillog-kafka:2222` host with SASL authentication.Make sure to adjust the paths, topic names, and authentication details according to your specific setup. Let me know if this worked. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [April 24, 2024, 1:08pm UTC](https://discuss.elastic.co/t/configure-two-sets-of-hosts-for-different-logs/358094/4 "2024-04-24T13:08:05Z")
> [@Adriann](#): > > ```auto > output.kafka: > hosts: > - syslog-kafka:2222 > - maillog-kafka:2222 > > ``` Did you test it? I don't think this works, in the `hosts` settings you put the hosts to where filebeat will connect when starting/restarting to get the cluster metadata, it will then use the returned response of the metadata to send the events. If you have 2 different clusters in the `hosts` configuration, filebeat will not connect to both of them because it is expected for them to be part of the same cluster, so it will talke with one of them and get the metadata for this cluster, the other cluster will be completely ignored. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [April 24, 2024, 1:08pm UTC](https://discuss.elastic.co/t/configure-two-sets-of-hosts-for-different-logs/358094/5 "2024-04-24T13:08:44Z")
> [@swtg](#): > > I am looking to configure 2 sets of kafka hosts to which different logs will be sent. I don't think this is possible using just the `kafka` output. But you can do that if you use the `logstash` output and configure your logstash pipeline to send the data to different kafka clusters according to your needs. ---
**Author:** ![Adriann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adriann/32/77780_2.png) [@Adriann](https://discuss.elastic.co/u/Adriann)\ **Post date:** [April 24, 2024, 8:44pm UTC](https://discuss.elastic.co/t/configure-two-sets-of-hosts-for-different-logs/358094/6 "2024-04-24T20:44:47Z")
@leandrojmp you are this. This would not work as I thought. This solution need two instances of filebeat and logstash instance that would separate the results. ---
**Author:** ![swtg](https://avatars.discourse-cdn.com/v4/letter/s/bbce88/32.png) [@swtg](https://discuss.elastic.co/u/swtg)\ **Post date:** [April 25, 2024, 10:19pm UTC](https://discuss.elastic.co/t/configure-two-sets-of-hosts-for-different-logs/358094/7 "2024-04-25T22:19:37Z")
@leandrojmp I believe yes we are using logstash pipelines for both of them and kafka clusters is configured. I have edited the actual hosts details and just gave an example. In actual configurations we have got cluster of servers. Can you please help me with the configurations which is required for your suggested option. ---
**Author:** ![swtg](https://avatars.discourse-cdn.com/v4/letter/s/bbce88/32.png) [@swtg](https://discuss.elastic.co/u/swtg)\ **Post date:** [April 25, 2024, 10:25pm UTC](https://discuss.elastic.co/t/configure-two-sets-of-hosts-for-different-logs/358094/8 "2024-04-25T22:25:01Z")
> [@Adriann](#): > > Please edit the post and format properly the configuration. @Adriann Sorry I am new to the forum and couldn't find the edit post option, however below is the correct format: ```bash filebeat.inputs: - type: syslog format: auto filebeat.inputs: - type: filestream id: maillog enabled: true paths: - /var/log/maillog output.kafka: topic: topic-syslog hosts: - syslog-kafka:2222 topics: - topic: topic-syslog when.contains: fileset.name: "syslog" output.kafka: topic: topic-maillog sasl.mechanism: PLAIN username: "abc" password: "xx" hosts: - maillog-kafka:2222 topics: - topic: topic-maillog when.contains: input.id: "maillog" ``` ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [April 26, 2024, 1:13am UTC](https://discuss.elastic.co/t/configure-two-sets-of-hosts-for-different-logs/358094/9 "2024-04-26T01:13:29Z")
> [@swtg](#): > > I believe yes we are using logstash pipelines for both of them and kafka clusters is configured. I have edited the actual hosts details and just gave an example. You need to share your logstash configurations. > [@swtg](#): > > ```auto > output.kafka: > topic: topic-syslog > hosts: > - syslog-kafka:2222 > topics: > - topic: topic-syslog > when.contains: > fileset.name: "syslog" > > output.kafka: > topic: topic-maillog > sasl.mechanism: PLAIN > username: "abc" > password: "xx" > hosts: > - maillog-kafka:2222 > topics: > - topic: topic-maillog > when.contains: > input.id: "maillog" > > ``` This does not work, you can have only **one** output block. ---
**Author:** ![swtg](https://avatars.discourse-cdn.com/v4/letter/s/bbce88/32.png) [@swtg](https://discuss.elastic.co/u/swtg)\ **Post date:** [April 26, 2024, 3:23am UTC](https://discuss.elastic.co/t/configure-two-sets-of-hosts-for-different-logs/358094/10 "2024-04-26T03:23:27Z")
> [@leandrojmp](#): > > You need to share your logstash configurations. Thanks for the reply! Unfortunately I dont have access to the server side and I am quite new to Elastic. Can you please let me know what exactly are you after, I can ask the team about it.