Logs from different ports issue

Elastic Stack Logstash
1

Hello,

I have the following setup:

  • system logs are sent to logstash using filebeat on port 5044,
  • Apache logs being sent to logstash (same logstash instance as above) using filebeat on port 5047

I then pull these logs as follows:
/etc/logstash/conf.d/system.conf

input {
  beats {
    port => 5044
  }
}
filter {}
output {
elasticsearch {
     hosts => ["https://localhost:9200"]
     cacert => "/etc/logstash/conf.d/es-ca.crt"
     index => "system"
     user => "filebeat"
     password => "password"
     }
   }

/etc/logstash/conf.d/apache.conf

input {
  beats {
    port => 5047
  }
}
filter {}
output {
elasticsearch {
     hosts => ["https://localhost:9200"]
     cacert => "/etc/logstash/conf.d/es-ca.crt"
     index => "apache"
     user => "filebeat"
     password => "password"
     }
   }

The issue I have is that, as much as the logs are received, they are mixed. Logs that should be in port 5047 are found on 5044, and vice versa.

How can I ensure logs hitting port 5044 should not also hit 5047?

2

You need to configure logstash to use multiple pipelines.

Per default it will merge all config files in the /etc/logstash/conf.d path, so you have just one config where all input will be sent to all filters and output, unless you use conditionals.

If the pipelines needs to be separated from each other, then the best approach is to configure multiple pipelines in pipelines.yml.

Change your pipelines.yml to this one and restart the logstash service.

- pipeline.id: system
  path.config: "/etc/logstash/conf.d/system.conf"
- pipeline.id: apache
  path.config: "/etc/logstash/conf.d/apache.conf"
3

Thanks,

I was using this approach in a different scenario. I will use it when needed.

Regards