# Kafka logstash logs are showing into filebeat logstash index **URL:** \ **Category:** Logstash\ **Created:** [July 3, 2023, 6:46am UTC](https://discuss.elastic.co/t/kafka-logstash-logs-are-showing-into-filebeat-logstash-index/337419 "2023-07-03T06:46:28Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![lalchand\_rajak](https://avatars.discourse-cdn.com/v4/letter/l/8e7dd6/32.png) [@lalchand\_rajak](https://discuss.elastic.co/u/lalchand_rajak)\ **Post date:** [July 3, 2023, 6:46am UTC](https://discuss.elastic.co/t/kafka-logstash-logs-are-showing-into-filebeat-logstash-index/337419/1 "2023-07-03T06:46:28Z")
Hello, I have kafka-logstash conf and logstash reciveing the logs from kafka. here is the config. ```auto input { kafka { topics => ["sitlogtopic","locallogtopic"] bootstrap_servers => "ddr-kafkadev.pvt.ccilindia.com:9092" #auto_offset_reset => "earliest" #consumer_threads => 1 #decorate_events => true } } filter{ grok { match => { "message" => ["%{DATESTAMP:Timestamp} %{LOGLEVEL:Loglevel} %{WORD:hostname} (?%{WORD}\-%{WORD}) %{GREEDYDATA:MESSAGE}", "%{DATESTAMP:Timestamp} %{LOGLEVEL:Loglevel} %{WORD:hostname} %{WORD:SpringAppName} %{GREEDYDATA:MESSAGE}","%{DATESTAMP:Timestamp} %{LOGLEVEL:Loglevel} %{WORD:HOSTNAME} (?%{WORD}\ %{WORD}) %{GREEDYDATA:MESSAGE}","%{DATESTAMP:Timestamp} %{LOGLEVEL:Loglevel} (?%{WORD}\-%{WORD}\-%{WORD}) %{WORD:SpringAppName} %{GREEDYDATA:MESSAGE}","%{DATESTAMP:Timestamp} %{LOGLEVEL:Loglevel} (?%{WORD}\-%{WORD}\-%{WORD}) (?%{WORD}\ %{WORD}) %{GREEDYDATA:MESSAGE}","%{DATESTAMP:Timestamp} %{LOGLEVEL:Loglevel} (?%{WORD}\-%{WORD}\-%{WORD}) %{WORD:SpringAppName} %{GREEDYDATA:MESSAGE}","%{DATESTAMP:Timestamp} %{LOGLEVEL:Loglevel} (?%{WORD}\-%{WORD}\-%{WORD}\-%{WORD}\-%{WORD}\-%{WORD}) (?%{WORD}\ %{WORD}) %{GREEDYDATA:MESSAGE}","(?%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}) %{LOGLEVEL:level} %{WORD:hostname} (?%{WORD}) (?(.|\r|\n)*)","%{DATESTAMP:Timestamp} %{LOGLEVEL:Loglevel} (?%{WORD}\-%{WORD}\-%{WORD}\-%{WORD}\-%{WORD}\-%{WORD}) (?%{WORD}\ %{WORD}) %{GREEDYDATA:MESSAGE}","%{DATESTAMP:Timestamp} %{LOGLEVEL:Loglevel} (?%{WORD}\-%{WORD}) (?%{WORD}\ %{WORD}) %{GREEDYDATA:MESSAGE}","%{DATESTAMP:Timestamp} %{LOGLEVEL:Loglevel} %{WORD:hostname} (?%{WORD}\-%{WORD}) (?(.|\r|\n)*)","%{DATESTAMP:Timestamp} %{LOGLEVEL:Loglevel} (?%{WORD}\-%{WORD}\-%{WORD}\-%{WORD}\-%{WORD}\-%{WORD}) (?%{WORD}\-%{WORD}) %{GREEDYDATA:MESSAGE}","%{DATESTAMP:Timestamp} %{LOGLEVEL:Loglevel} (?%{WORD}\-%{WORD}-%{WORD}) (?%{WORD}\-%{WORD}) %{GREEDYDATA:MESSAGE}"]} } #mutate { add_field => { "SpringAppName" => "%{Ap}%{Service}" } } mutate { remove_field => ['message','event.original'] } } output { elasticsearch { hosts => ["https://elastic-uat.ccilindia.net:9200"] #index => "kafkadev-%{+yyyy.MM.dd}" ilm_rollover_alias => "kafkadev" ilm_pattern => "000001" ilm_policy => "kafkadev" ilm_enabled => true cacert => "/etc/logstash/certs/GeoTrust-RSA-CA-Intermediate-2018.pem" user => "elastic" password => "Ccil@2023" ssl => true ssl_certificate_verification => true } stdout { codec => rubydebug } } ``` but I have another server where I am sending filebeat logs to logstash but i am seeing the same kafka logs on filebeat index . here the filbeat logstash config. ```auto input { beats { port => 5044 } } output { elasticsearch { hosts => ["https://elastic-uat.ccilindia.net:9200"] user => "elastic" password => "Ccil@2023" cacert => "/etc/logstash/certs/GeoTrust-RSA-CA-Intermediate-2018.pem" index => "gitlab-filebeat-8.7" ssl => true ssl_certificate_verification => false } stdout { codec =>rubydebug } } ``` ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [July 3, 2023, 12:57pm UTC](https://discuss.elastic.co/t/kafka-logstash-logs-are-showing-into-filebeat-logstash-index/337419/2 "2023-07-03T12:57:10Z")
> [@lalchand\_rajak](#): > > I have another server where I am sending filebeat logs to logstash but i am seeing the same kafka logs on filebeat index This is another Logstash server or another Logstash pipeline on the same server? ---
**Author:** ![lalchand\_rajak](https://avatars.discourse-cdn.com/v4/letter/l/8e7dd6/32.png) [@lalchand\_rajak](https://discuss.elastic.co/u/lalchand_rajak)\ **Post date:** [July 3, 2023, 1:28pm UTC](https://discuss.elastic.co/t/kafka-logstash-logs-are-showing-into-filebeat-logstash-index/337419/3 "2023-07-03T13:28:12Z")
Logstash server is same but different logstash pipeline ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [July 3, 2023, 1:48pm UTC](https://discuss.elastic.co/t/kafka-logstash-logs-are-showing-into-filebeat-logstash-index/337419/4 "2023-07-03T13:48:20Z")
> [@lalchand\_rajak](#): > > Logstash server is same but different logstash pipeline How are you running logstash? As a service? What does your `pipelines.yml` looks like? ---
**Author:** ![lalchand\_rajak](https://avatars.discourse-cdn.com/v4/letter/l/8e7dd6/32.png) [@lalchand\_rajak](https://discuss.elastic.co/u/lalchand_rajak)\ **Post date:** [July 3, 2023, 2:47pm UTC](https://discuss.elastic.co/t/kafka-logstash-logs-are-showing-into-filebeat-logstash-index/337419/5 "2023-07-03T14:47:34Z")
yes as service. pipelines.yml ```auto cat pipelines.yml # This file is where you define your pipelines. You can define multiple. # For more information on multiple pipelines, see the documentation: # https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html - pipeline.id: main path.config: "/etc/logstash/conf.d/*.conf" ``` ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [July 3, 2023, 3:46pm UTC](https://discuss.elastic.co/t/kafka-logstash-logs-are-showing-into-filebeat-logstash-index/337419/6 "2023-07-03T15:46:40Z")
> [@lalchand\_rajak](#): > > ```auto > - pipeline.id: main > path.config: "/etc/logstash/conf.d/*.conf" > > ``` That's the issue, you are running just one pipeline, Logstash will merge both your pipeline configurations and data from both inputs will be sent to both outputs. You need to configure `pipelines.yml` to use multiple pipelines as explained in the [documentation](https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html). This way your pipelines will be independent from each other. ---
**Author:** ![lalchand\_rajak](https://avatars.discourse-cdn.com/v4/letter/l/8e7dd6/32.png) [@lalchand\_rajak](https://discuss.elastic.co/u/lalchand_rajak)\ **Post date:** [July 3, 2023, 4:39pm UTC](https://discuss.elastic.co/t/kafka-logstash-logs-are-showing-into-filebeat-logstash-index/337419/8 "2023-07-03T16:39:28Z")
even after adding two pipeline id. still i can see the some logs from kafka index. here is pipeline.yml ```auto # This file is where you define your pipelines. You can define multiple. # For more information on multiple pipelines, see the documentation: # https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html #- pipeline.id: main # path.config: "/etc/logstash/conf.d/*.conf" - pipeline.id: kafka path.config: "/etc/logstash/conf.d/kafkasre.conf" - pipeline.id: gitlab path.config: "/etc/logstash/conf.d/gitlab.conf" ``` ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [July 3, 2023, 5:18pm UTC](https://discuss.elastic.co/t/kafka-logstash-logs-are-showing-into-filebeat-logstash-index/337419/9 "2023-07-03T17:18:33Z")
> [@lalchand\_rajak](#): > > even after adding two pipeline id. still i can see the some logs from kafka index. here is pipeline.yml Are you still seeing new logs from the Kafka input in the beats indice after you restarted Logstash? ---
**Author:** ![lalchand\_rajak](https://avatars.discourse-cdn.com/v4/letter/l/8e7dd6/32.png) [@lalchand\_rajak](https://discuss.elastic.co/u/lalchand_rajak)\ **Post date:** [July 3, 2023, 5:35pm UTC](https://discuss.elastic.co/t/kafka-logstash-logs-are-showing-into-filebeat-logstash-index/337419/10 "2023-07-03T17:35:40Z")
Yes, post logstash service restart ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [July 3, 2023, 5:43pm UTC](https://discuss.elastic.co/t/kafka-logstash-logs-are-showing-into-filebeat-logstash-index/337419/11 "2023-07-03T17:43:45Z")
> [@lalchand\_rajak](#): > > Yes, post logstash service restart That's not correct, so something is still wrong in your configuration. Please restart your logstash to get fresh logs and share the logs it creates while starting. Also, run the following request in your logstash server and share the result: ```auto curl http://localhost:9600/_node/pipelines?pretty ``` ---
**Author:** ![lalchand\_rajak](https://avatars.discourse-cdn.com/v4/letter/l/8e7dd6/32.png) [@lalchand\_rajak](https://discuss.elastic.co/u/lalchand_rajak)\ **Post date:** [July 14, 2023, 2:27pm UTC](https://discuss.elastic.co/t/kafka-logstash-logs-are-showing-into-filebeat-logstash-index/337419/12 "2023-07-14T14:27:36Z")
I was able to solve the issue by commenting the config path file location in logstash.ynl ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 11, 2023, 2:28pm UTC](https://discuss.elastic.co/t/kafka-logstash-logs-are-showing-into-filebeat-logstash-index/337419/13 "2023-08-11T14:28:24Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.