Filebeat is running, but can't detect filelogs is changed

[Quang_Tho]

I am deploy filebeat for shipping docker logs to elk.
Filebeat is running but cannot ship log to logstash.
is any missing here, please check for me.

--- docker-compose.yml -----

version: "3.6"
services:
#################  WEB ###################
  nginx:
      image: nginx:1.14.0
      ports:
        - 80:80
      volumes:
        - type: volume
          source: nginx-config
          target: /etc/nginx
        - type: volume
          source: nginx-doc-root
          target: /usr/share/nginx/html
      deploy:
        replicas: 1
        placement:
          constraints:
           - node.labels.type == web

  filebeat:
    image: docker.elastic.co/beats/filebeat:6.3.2
    volumes:
      - type: volume
        source: filebeat-config
        target: /usr/share/filebeat/
    deploy:
      replicas: 1
      placement:
        constraints:
         - node.labels.type == web

################### MONITOR ###############
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.3.2
    ports:
      - 9200:9200
      - 9300:9300
    environment:
      ES_JAVA_OPTS: "-Xms512m -Xmx512m"
    volumes:
      - type: volume
        source: elasticsearch
        target:  /usr/share/elasticsearch/config/
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.labels.type == monitor

####  indexer
  logstash:
    image: docker.elastic.co/logstash/logstash:6.3.2
    ports:
      - 5044:5044
    volumes:
      - type: volume
        source: logstash-p
        target: /usr/share/logstash/pipeline/
      - type: volume
        source: logstash-config
        target: /usr/share/logstash/config/
    depends_on:
      - elasticsearch
    deploy:
      replicas: 1
      placement:
        constraints:
         - node.labels.type == monitor

###   UI
  kibana:
    image: docker.elastic.co/kibana/kibana:6.3.2
    ports:
      - 5601:5601
    environment:
      ELASTICSEARCH_URL: http://elasticsearch:9200
    depends_on:
      - elasticsearch
    volumes:
      - type: volume
        source: kibana
        target: /usr/share/kibana/config
    deploy:
      replicas: 1
      placement:
        constraints:
         - node.labels.type == monitor

volumes:
  elasticsearch:
  logstash-p:
  logstash-config:
  kibana:
  filebeat-config:
  nginx-config:
  nginx-doc-root:

Here are filebeat.yml for test.

filebeat.inputs:
- type: log
  paths:
    - /var/lib/docker/containers/*/*.log

logging.level: debug
logging.selectors: ["*"]

output.console:
  pretty: true

Some logs from Filebeat:

2018-08-07T07:11:58.774Z        INFO    [monitoring]    log/log.go:124  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":370,"time":{"ms":5}},"total":{"ticks":660,"time":{"ms":9},"value":660},"user":{"ticks":290,"time":{"ms":4}}},"info":{"ephemeral_id":"cb1e279d-16f2-4c1f-a72b-f0acaa155a84","uptime":{"ms":1530041}},"memstats":{"gc_next":4194304,"memory_alloc":1749184,"memory_total":12509528}},**"filebeat":{"harvester":{"open_files":0,"running":0}},**"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":2,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.04,"15":0.05,"5":0.06,"norm":{"1":0.01,"15":0.0125,"5":0.015}}}}}}

[jsoriano (Jaime Soriano)]

Hi @Quang_Tho and welcome

filebeat needs access to the log files, in your configuration the input path is set to /var/lib/docker/containers/*/*.log, that is a path in the host, but this path is not mounted as a volume in the filebeat docker.
You may also want to use the add_docker_metadata processor or the docker autodiscover provider, in that case you will also need to mount the docker socket.

Regarding shipping the logs to logstash, in your configuration the output is set to console only, if you want the logs to be shipped to logstash you need to configure a logstash output instead.

[Quang_Tho]

@jsoriano
it is working.
thanks you very much!