Filebeat send not all log files

Elastic Stack Beats
1

I got this Filebeat config:

I got this Filebeat config:

filebeat.prospectors:
- input_type: log
  paths:
- /var/lib/docker/containers/*/*.log
  document_type: docker
  json.message_key: log
output.elasticsearch:
  hosts: ["localhost:9200"]
setup.kibana:
  host: "localhost"
processors:
- add_docker_metadata:
host: "unix:///var/run/docker.sock"
match_fields: ["system.process.cgroup.id"]
match_pids: ["process.pid", "process.ppid"]
match_source: true
match_source_index: 4
match_short_id: true

So as you can see i try to grab all logs from docker containers. Actually i can see only logs only one container, other logs (i got 3 containers) not sending to logstash. Where is my mistake?

2

The indentation of the configuration seems to be off:

filebeat.prospectors:
- input_type: log
  enabled: true
  paths:
    - /var/lib/docker/containers/*/*.log
  document_type: docker
  json.message_key: log

output.elasticsearch:
  hosts: ["localhost:9200"]

setup.kibana:
  host: "localhost"

processors:
- add_docker_metadata:
    host: "unix:///var/run/docker.sock"
    match_fields: ["system.process.cgroup.id"]
    match_pids: ["process.pid", "process.ppid"]
    match_source: true
    match_source_index: 4
    match_short_id: true

Also the log input is not enabled. If the input is not enabled it is not supposed to read events. Have you tried using the docker input instead? https://www.elastic.co/guide/en/beats/filebeat/6.5/filebeat-input-docker.html
It contains Docker specific options. It is superior compared to log when it comes to Docker logs.

You mentioned that you are sending to Logstash. But in the configuration you have shared the output is Elasticsearch. Are you sure that it is the configuration you are runnning?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.