Filebeat ignoring Path with Wildcard(s)

[allenfin (Allen)]

I'm using Filebeat as a docker container and I keep having problems with trying make sure that nginx error logs and access logs go to the right place. It seems that they just go to both places. I setup a different logstash port for each one.

Here's how I launch a container:

sudo docker run -d -v /path/to/nginx/logs:/var/log/nginx -v /path/to/filebeats/nginx_access.yml:/usr/share/filebeat/filebeat.yml --name nginx_access docker.elastic.co/beats/filebeat:5.4.2;

Here's how my filebeat.yml file is:

filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/nginx/access*.log
  exclude_files: [".gz$"]
output.logstash:
  hosts: ["host:9251"]

I've also tried using /var/log/nginx/access*.log* as one of the paths, but each filebeat container will still read all of the log files. Am i doing something wrong? Should I upgrade the container to the latest version?

[exekias (Carlos Pérez Aradros)]

Hi @allenfin,

What do you mean by sending them to both places?

In this particular case it seems you are trying to retrieve NGINX logs, we have a module for that that should ease your setup process! Take a look to https://www.elastic.co/guide/en/beats/filebeat/5.4/_tutorial.html, probably all you want to have is:

filebeat.modules:
- module: nginx
output.logstash:
  hosts: ["host:9251"]

[allenfin (Allen)]

Sorry, I have setup 2 filebeats. 1 for nginx access and another for error. Here is the second one:

filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/nginx/error*.log
  exclude_files: [".gz$"]
output.logstash:
  hosts: ["host:9250"]

They are using different logstash processes, so they are on different ports. When I check kibana, it shows that the same lines are in 2 different indexes defined for each different logstash port. And the source lists the same file, so this would mean it's just reading all of the logs instead of following the paths I have define.