Don't see the metrics I expect

Hi there,
I'm trying to get some Fargate metrics using Cloudwatch namespaces, but I don't see a way to get them to Kibana.
modules.d/aws.yml

    - module: aws
      period: 5m
      access_key_id: 'xxxxxxxxxxxxx'
      secret_access_key: 'xxxxxxxxxxxxxxxxxxxxxxx'
      metricsets:
        - rds
        - usage
        - ebs
        - ec2
        - cloudwatch
      metrics:
        - namespace: "*"

in the logs output I can see this, but not sure how to check this in kibana:

      "aws": {
        "ecs": {
          "metrics": {
            "MemoryUtilization": {
              "avg": 15.336588541666666,
              "max": 15.4296875,
              "min": 15.234375,
              "sum": 153.36588541666666,
              "count": 10
            },
            "CPUUtilization": {
              "sum": 5.197281962927915,
              "count": 10,
              "avg": 0.5197281962927915,
              "max": 1.1245832656975834,
              "min": 0.2879062617197633
            }
          }
        },
        "cloudwatch": {
          "namespace": "AWS/ECS"
        },
        "dimensions": {
          "ServiceName": "web-service",
          "ClusterName": "tt-ecs"
        }
      },
      "event": {
        "dataset": "aws.cloudwatch",
        "module": "aws",
        "duration": 120682193
      },
      "metricset": {
        "name": "cloudwatch",
        "period": 300000
      }
    }

How would I be able to visualize these metrics?

Did you integrate filebeat with Elasticsearch? See: Filebeat quick start: installation and configuration | Filebeat Reference [master] | Elastic

Hi Marcin,
so you mean I should install filebeat inside the containers running on ECS? I've installed firelens container with fluentbit as AWS recommends for log streaming to fluentd.

Could you please post configuration for filebeat? I don't see where did you configure the connection to Elasticsearch.

not using filebeat but metricbeat to get the metrics from AWS Cloudwatch AWS/ContainerInsights namespace.
Metricbeat configuration is pretty simple,

metricbeat.config.modules:

  path: ${path.config}/modules.d/*.yml

reload.enabled: yes

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression

setup.kibana:
output.elasticsearch:
  hosts: ["localhost:9200"]

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
  - add_observer_metadata: ~

logging.level: debug
logging.to_files: true
logging.files:
  path: /var/log/metricbeat
  name: metricbeat
  keepfiles: 7
  permissions: 0644

Sorry misread the fact that you're using metricbeat. Actually this is the snippet I was looking for:

output.elasticsearch:
  hosts: ["localhost:9200"]

Could you please check if any data land in the ES cluster? As you said you can see outputted logs (process stdout?)

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.