Apache Dashboards blank for Docker containers

[castle9mm]

Hello, I'm really struggling and could use some help. I really don't even know where to begin. I'm trying to setup Elasticsearch and Kibana with Filebeat, all as docker containers, to read apache logs inside different docker containers.

I can see the beat log in 'Discover' and the apache access line is stored in the 'log' variable, however, the dashboard seems to be looking for specifics like 'apache2.access.response_code'. How do I process the apache 'log' message into the correct apache variables for use in the dashboard?

I'm mounting the /var/lib/docker/containers on the host directly into the filebeat container to read, as well as the /var/run/docker.sock

Can anyone point me in the right direction? Thank you in advance for any help.

Here is my filebeat.yml:

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

filebeat.prospectors:
- input_type: log
  paths:
    - /var/lib/docker/containers/*/*.log
  document_type: docker
  json.message_key: log

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains:
              docker.container.image: "mywordpress"
          config:
            - module: apache2
              access:
                prospector:
                  type: docker
                  container.ids:
                    - "${data.docker.container.id}"

processors:
- add_cloud_metadata:

output.elasticsearch:
  hosts: ['LB.ELK.rancher.internal:9200']

[exekias (Carlos Pérez Aradros)]

Hi @castle9mm,

There is a typo in your settings, it should be containers.ids, instead of container.ids. Also take into account that you can use docker prospector for the general case too, this is how the final config would look:

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

filebeat.prospectors:
- type: docker
  containers.ids:
    - '*'
  document_type: docker

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains:
              docker.container.image: "mywordpress"
          config:
            - module: apache2
              access:
                prospector:
                  type: docker
                  containers.ids:
                    - "${data.docker.container.id}"

processors:
- add_cloud_metadata:

output.elasticsearch:
  hosts: ['LB.ELK.rancher.internal:9200']

[castle9mm]

Thanks for that Carlos. I made the changes but still nothing for apache. I do see new "prospector.type docker" on the Discover page so it does seem like it's picking up some containers at least. This is the filebeat log when I restart the apache container. Anything stand out?

I do see autodiscover starting an apache runner after starting the filebeat container but then there's an error when I restart the apache container.

1/25/2018 11:20:37 AM2018/01/25 18:20:37.155433 cfgwarn.go:23: WARN EXPERIMENTAL: Docker prospector is enabled.
1/25/2018 11:20:37 AM2018/01/25 18:20:37.157290 autodiscover.go:138: INFO Autodiscover starting runner: apache2 (error, access)
1/25/2018 11:20:37 AM2018/01/25 18:20:37.157452 client.go:123: INFO Elasticsearch url: http://LB.ELK.rancher.internal:9200
1/25/2018 11:20:37 AM2018/01/25 18:20:37.159052 reload.go:258: INFO Starting 2 runners ...

1/25/2018 11:20:37 AM2018/01/25 18:20:37.171137 prospector.go:87: INFO Starting prospector of type: log; ID: 17216501560277081620
1/25/2018 11:20:37 AM2018/01/25 18:20:37.171165 prospector.go:87: INFO Starting prospector of type: log; ID: 4128373258093383538
1/25/2018 11:20:37 AM2018/01/25 18:20:37.171380 client.go:123: INFO Elasticsearch url: http://LB.ELK.rancher.internal:9200
1/25/2018 11:20:37 AM2018/01/25 18:20:37.172190 prospector.go:87: INFO Starting prospector of type: docker; ID: 7236295744761858416
1/25/2018 11:20:37 AM2018/01/25 18:20:37.172217 prospector.go:87: INFO Starting prospector of type: log; ID: 4128373258093383538
1/25/2018 11:20:37 AM2018/01/25 18:20:37.172649 harvester.go:215: INFO Harvester started for file: /var/lib/docker/containers/462de10287a5c04facac093b977fb64a6bfedf6f30f68008dd7bdbfdea564a4e/462de10287a5c04facac093b977fb64a6bfedf6f30f68008dd7bdbfdea564a4e-json.log
1/25/2018 11:20:37 AM2018/01/25 18:20:37.174814 client.go:651: INFO Connected to Elasticsearch version 6.1.2
1/25/2018 11:20:37 AM2018/01/25 18:20:37.184639 prospector.go:87: INFO Starting prospector of type: log; ID: 16372112168949479980
1/25/2018 11:20:37 AM2018/01/25 18:20:37.184677 prospector.go:87: INFO Starting prospector of type: log; ID: 5620371616911475983
1/25/2018 11:20:37 AM2018/01/25 18:20:37.184723 reload.go:219: INFO Loading of config files completed.

1/25/2018 11:24:04 AM2018/01/25 18:24:04.111793 metrics.go:39: INFO Non-zero metrics in the last 30s: beat.info.uptime.ms=30000 beat.memstats.gc_next=20843600 beat.memstats.memory_alloc=14920472 beat.memstats.memory_total=1710144384 filebeat.events.added=27474 filebeat.events.done=27474 filebeat.harvester.open_files=20 filebeat.harvester.running=20 libbeat.config.module.running=2 libbeat.output.read.bytes=424484 libbeat.output.write.bytes=15652581 libbeat.pipeline.clients=11 libbeat.pipeline.events.active=4117 libbeat.pipeline.events.published=27474 libbeat.pipeline.events.total=27474 libbeat.pipeline.queue.acked=27474 registrar.states.current=19 registrar.states.update=27474 registrar.writes=550
1/25/2018 11:24:10 AM2018/01/25 18:24:10.268224 autodiscover.go:175: INFO Autodiscover stopping runner: apache2 (access, error)
1/25/2018 11:24:10 AM2018/01/25 18:24:10.268636 prospector.go:121: INFO Prospector ticker stopped
1/25/2018 11:24:10 AM2018/01/25 18:24:10.268659 prospector.go:138: INFO Stopping Prospector: 7236295744761858416
1/25/2018 11:24:10 AM2018/01/25 18:24:10.268716 harvester.go:236: INFO Reader was closed: /var/lib/docker/containers/462de10287a5c04facac093b977fb64a6bfedf6f30f68008dd7bdbfdea564a4e/462de10287a5c04facac093b977fb64a6bfedf6f30f68008dd7bdbfdea564a4e-json.log. Closing.
1/25/2018 11:24:10 AM2018/01/25 18:24:10.268794 prospector.go:121: INFO Prospector ticker stopped
1/25/2018 11:24:10 AM2018/01/25 18:24:10.268817 prospector.go:138: INFO Stopping Prospector: 4128373258093383538
1/25/2018 11:24:12 AM2018/01/25 18:24:12.722325 cfgwarn.go:23: WARN EXPERIMENTAL: Docker prospector is enabled.
1/25/2018 11:24:12 AM2018/01/25 18:24:12.722882 factory.go:69: ERR Error creating prospector: Can only start a prospector when all related states are finished: {Id: Finished:false Fileinfo:0xc4203be680 Source:/var/lib/docker/containers/462de10287a5c04facac093b977fb64a6bfedf6f30f68008dd7bdbfdea564a4e/462de10287a5c04facac093b977fb64a6bfedf6f30f68008dd7bdbfdea564a4e-json.log Offset:4812082 Timestamp:2018-01-25 18:24:12.698948088 +0000 UTC m=+218.599024642 TTL:-1ns Type:docker FileStateOS:5249732-64768}

[castle9mm]

Actually, I found the issue. I was filtering the Discover page based on container image but the filebeat.yml from Carlos didn't have the docker metadata added. So the logs were there but not showing in my filter. I modified my filebeat.yml to include the docker metadata on the prospectors and i'm seeing data in the Apache2 dashboards now and my filters are working. Thank you all! Posted below is is my working config.

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

filebeat.prospectors:
- type: docker
  containers.ids:
    - '*'
  document_type: docker
  processors:
    - add_docker_metadata: ~

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains:
              docker.container.image: "castle9mm/mywordpress"
          config:
            - module: apache2
              access:
                prospector:
                  type: docker
                  containers.ids:
                    - "${data.docker.container.id}"

processors:
- add_cloud_metadata:

output.elasticsearch:
  hosts: ['LB.ELK.rancher.internal:9200']

setup.kibana:
  host: "LB.ELK.rancher.internal:5601"