Ecs, metricbeat, docker, apache, stdout where am I going wrong?

[ppafford (Phill Pafford)]

Running metricbeat on ECS cluster in it's own docker container. running my application (including apache with the application) on another container. Both my app and apache log to STDOUT. have metricbeats setup for auto discovery but no apache logs are coming in.

metricbeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
    - type: docker
      templates:
        - condition:
            contains:
              docker.container.labels.applicationModules: "apache"
          config:
            - module: apache
              metricsets: ["status"]
              period: 10s
              hosts: ["${data.host}:${data.port}"]

I set this label on my docker container when starting it up using the ecs task. I can see some log info but it's the format. example message

{
  "_index": "metricbeat-6.5.4-2019.03.14",
  "_type": "doc",
  "_id": "asdfsdkjfsk",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-03-14T19:55:00.791Z",
    "docker": {
      "container": {
        "id": "asdjkhkj3n4rj3n4nrksdjcow4r",
        "labels": {
          "environment": "dev",
          "applicationModules": "apache",
          "applicationName": "myapp"
        }
      }
    },
    "beat": {
      "name": "Elastic-Metricbeat.weave.local",
      "hostname": "Elastic-Metricbeat.weave.local",
      "version": "6.5.4"
    },
    "host": {
      "name": "Elastic-Metricbeat.weave.local"
    },
    "metricset": {
      "host": "121.44.0.56:443",
      "rtt": 4128,
      "name": "status",
      "module": "apache"
    },
    "error": {
      "message": "HTTP error 400 in status: 400 Bad Request"
    }
  },
  "fields": {
    "@timestamp": [
      "2019-03-14T19:55:00.791Z"
    ]
  },
  "highlight": {
    "docker.container.labels.applicationName": [
      "@kibana-highlighted-field@alert-api@/kibana-highlighted-field@"
    ],
    "metricset.module": [
      "@kibana-highlighted-field@apache@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1552593300791
  ]
}

looking more at the data it seems like Im only getting metrics from metricbeat, how do I get them from the other docker containers running?

[ruflin]

Here is a good blog post on how to get started with docker and metricbeat: https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-docker.html

I do not really know Amazon ECS. The part I'm wondering is how you configured Metricbeat to have access to the docker host through the socket? Is that even possible in AWS ECS?

[ppafford (Phill Pafford)]

I'm trying to follow the link you sent. here is the task definition, it's running as a daemon on each ecs host but only in 1 docker container. so the idea is to connect to each host in the ecs cluster to grab all the application logs/metrics for any containers running specifically on that host.

{
  "ipcMode": null,
  "executionRoleArn": null,
  "containerDefinitions": [
    {
      "dnsSearchDomains": null,
      "logConfiguration": {
        "logDriver": "json-file",
        "options": {
          "labels": "applicationName,environment"
        }
      },
      "entryPoint": null,
      "portMappings": [],
      "command": null,
      "linuxParameters": null,
      "cpu": 10,
      "environment": [],
      "resourceRequirements": null,
      "ulimits": null,
      "mountPoints": [
        {
          "readOnly": true,
          "containerPath": "/var/run/docker.sock",
          "sourceVolume": "docker_sock"
        },
        {
          "readOnly": true,
          "containerPath": "/hostfs/sys/fs/cgroup",
          "sourceVolume": "cgroup"
        },
        {
          "readOnly": true,
          "containerPath": "/hostfs/proc",
          "sourceVolume": "proc"
        },
        {
          "readOnly": true,
          "containerPath": "/hostfs",
          "sourceVolume": "root"
        }
      ],
      "volumes": [
        {
          "name": "root",
          "host": {
            "sourcePath": "/"
          },
          "dockerVolumeConfiguration": null
        },
        {
          "name": "proc",
          "host": {
            "sourcePath": "/proc"
          },
          "dockerVolumeConfiguration": null
        },
        {
          "name": "cgroup",
          "host": {
            "sourcePath": "/sys/fs/cgroup"
          },
          "dockerVolumeConfiguration": null
        },
        {
          "name": "docker_sock",
          "host": {
            "sourcePath": "/var/run/docker.sock"
          },
          "dockerVolumeConfiguration": null
        }
      ],
      "workingDirectory": null,
      "secrets": null,
      "dockerSecurityOptions": null,
      "memory": 256,
      "memoryReservation": null,
      "volumesFrom": [],
      "stopTimeout": null,
      "startTimeout": null,
      "dependsOn": null,
      "disableNetworking": null,
      "interactive": null,
      "healthCheck": null,
      "essential": true,
      "links": null,
      "hostname": "my-app.local",
      "extraHosts": null,
      "pseudoTerminal": null,
      "user": null,
      "readonlyRootFilesystem": null,
      "dockerLabels": {
        "environment": "dev",
        "applicationName": "metricbeat",
      },
      "systemControls": null,
      "privileged": null,
      "name": "Elastic-Metricbeat"
    }
  ],
  "placementConstraints": [],
  "memory": null,
  "compatibilities": [
    "EC2"
  ],
  "family": "Elastic-Metricbeat",
  "requiresAttributes": [
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.17"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.ecr-auth"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.task-iam-role"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.logging-driver.json-file"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
    }
  ],
  "pidMode": null,
  "requiresCompatibilities": [],
  "networkMode": null,
  "cpu": null,
  "revision": 2,
  "status": "ACTIVE",
  "proxyConfiguration": null
}

[ppafford (Phill Pafford)]

my application is php, we use monolog for logs and I have it configured to log to STDOUT as well as a file

monolog:
    channels: [main]
    handlers:
        main:
            type:   rotating_file
            path:   "%kernel.logs_dir%/%kernel.environment%.log"
            level:  "%monolog_verbosity%"
            channels: [main, request, php]
            max_files: 30
        stdout:
          type:   stream
          path:   "php://stdout"
          level:  "%monolog_verbosity%"
          channels: [main, request, php]

apache configured to log to STDOUT as well

DocumentRoot /var/www/my-app

    SSLEngine on

    SSLCertificateFile  /path/my-app.crt
    SSLCertificateKeyFile /path/my-app.key

    ServerSignature Off
    SetEnvIf Remote_Addr "::1" dontlog
    LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined_with_x_forward_for

    CustomLog /proc/self/fd/1 combined_with_x_forward_for env=!dontlog
    ErrorLog /proc/self/fd/2

https://docs.docker.com/config/containers/logging/ point to here https://github.com/docker-library/httpd/blob/b13054c7de5c74bbaa6d595dbe38969e6d4f860c/2.2/Dockerfile#L72-L75 for Apache STDOUT