AWS Cloudwatch

Elastic Stack Beats
#1

SO: Ubuntu 20.04 LTS
metricbeat: metricbeat version 7.9.2 (amd64), libbeat 7.9.2 [2ab907f built 2020-09-22 23:25:17 +0000 UTC]

Hello there. I hope you're doing well.

I'm trying to set up a metricbeat agent. It will be running on an EC2 instance. But, I'm running it on localhost first. I'm using the DEB package. My goal is to get the cloudwatch metrics (There are EC2 metrics on my account) and send them to the ELK cluster. The system module is disabled and the AWS module is enabled. The agent can reach the cluster but I can't see metrics on Kibana. I don't see errors in the Syslog. Here are my files:

metricbeat.yml

# Module: aws
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/7.9/metricbeat-module-aws.html

- module: aws
  period: 5m
  access_key_id: '<<access_key>>'
  secret_access_key: '<<secret_key>>'
  metricsets:
    - cloudwatch
  metrics:
    - namespace: AWS/EC2
      name: ["CPUUtilization", "DiskWriteOps"]
      #resource_type: ec2:instance
      #dimensions:
      #  - name: InstanceId
      #    value: i-0686946e22cf9494a
      #statistic: ["Average", "Maximum"]

aws.yml

metricbeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  #_source.enabled: false

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

# =============================== Elastic Cloud ================================


cloud.id: "elastic-observability-deployment:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
cloud.auth: "elastic:XXXXXXXXXXXXXXXXX"


# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

Thank you so much !

#2

What's in your Metricbeat logs?

#3

This is a part of the log. Check the Write alias successfully generated.

     Sep 30 19:40:09 user metricbeat[40005]: 2020-09-30T19:40:09.473-0500#011INFO#011[index-management]#011idxmgmt/std.go:184#011Set output.elasticsearch.index to 'metricbeat-7.9.2' as ILM is enabled.
        Sep 30 19:40:09 user metricbeat[40005]: 2020-09-30T19:40:09.473-0500#011INFO#011eslegclient/connection.go:99#011elasticsearch url: https://4a11cf6113d94bffbf45eabb85ab2adf.us-east-1.aws.found.io:443
        Sep 30 19:40:09 user metricbeat[40005]: 2020-09-30T19:40:09.473-0500#011INFO#011[publisher]#011pipeline/module.go:113#011Beat name: user
        Sep 30 19:40:09 user metricbeat[40005]: 2020-09-30T19:40:09.492-0500#011INFO#011instance/beat.go:450#011metricbeat start running.
        Sep 30 19:40:09 user metricbeat[40005]: 2020-09-30T19:40:09.492-0500#011INFO#011[monitoring]#011log/log.go:118#011Starting metrics logging every 30s
        Sep 30 19:40:11 user metricbeat[40005]: 2020-09-30T19:40:11.542-0500#011INFO#011cfgfile/reload.go:164#011Config reloader started
        Sep 30 19:40:12 user metricbeat[40005]: 2020-09-30T19:40:12.441-0500#011INFO#011[add_cloud_metadata]#011add_cloud_metadata/add_cloud_metadata.go:89#011add_cloud_metadata: hosting provider type not detected.
        Sep 30 19:40:13 user metricbeat[40005]: 2020-09-30T19:40:13.206-0500#011INFO#011cfgfile/reload.go:224#011Loading of config files completed.
        Sep 30 19:40:30 user metricbeat[40005]: 2020-09-30T19:40:30.348-0500#011INFO#011[publisher]#011pipeline/retry.go:219#011retryer: send unwait signal to consumer
        Sep 30 19:40:30 user metricbeat[40005]: 2020-09-30T19:40:30.348-0500#011INFO#011[publisher]#011pipeline/retry.go:223#011  done
        Sep 30 19:40:30 user metricbeat[40005]: 2020-09-30T19:40:30.348-0500#011INFO#011[publisher_pipeline_output]#011pipeline/output.go:143#011Connecting to backoff(elasticsearch(https://4a11cf6113d94bffbf45eabb85ab2adf.us-east-1.aws.found.io:443))
        Sep 30 19:40:30 user metricbeat[40005]: 2020-09-30T19:40:30.914-0500#011INFO#011[esclientleg]#011eslegclient/connection.go:314#011Attempting to connect to Elasticsearch version 7.9.2
        Sep 30 19:40:31 user metricbeat[40005]: 2020-09-30T19:40:31.000-0500#011INFO#011[license]#011licenser/es_callback.go:51#011Elasticsearch license: Platinum
        Sep 30 19:40:31 user metricbeat[40005]: 2020-09-30T19:40:31.084-0500#011INFO#011[esclientleg]#011eslegclient/connection.go:314#011Attempting to connect to Elasticsearch version 7.9.2
        Sep 30 19:40:31 user metricbeat[40005]: 2020-09-30T19:40:31.176-0500#011INFO#011[index-management]#011idxmgmt/std.go:261#011Auto ILM enable success.
        Sep 30 19:40:31 user metricbeat[40005]: 2020-09-30T19:40:31.264-0500#011INFO#011[index-management.ilm]#011ilm/std.go:139#011do not generate ilm policy: exists=true, overwrite=false
        Sep 30 19:40:31 user metricbeat[40005]: 2020-09-30T19:40:31.264-0500#011INFO#011[index-management]#011idxmgmt/std.go:274#011ILM policy successfully loaded.
        Sep 30 19:40:31 user metricbeat[40005]: 2020-09-30T19:40:31.264-0500#011INFO#011[index-management]#011idxmgmt/std.go:407#011Set setup.template.name to '{metricbeat-7.9.2 {now/d}-000001}' as ILM is enabled.
        Sep 30 19:40:31 user metricbeat[40005]: 2020-09-30T19:40:31.265-0500#011INFO#011[index-management]#011idxmgmt/std.go:412#011Set setup.template.pattern to 'metricbeat-7.9.2-*' as ILM is enabled.
        Sep 30 19:40:31 user metricbeat[40005]: 2020-09-30T19:40:31.265-0500#011INFO#011[index-management]#011idxmgmt/std.go:446#011Set settings.index.lifecycle.rollover_alias in template to {metricbeat-7.9.2 {now/d}-000001} as ILM is enabled.
        Sep 30 19:40:31 user metricbeat[40005]: 2020-09-30T19:40:31.265-0500#011INFO#011[index-management]#011idxmgmt/std.go:450#011Set settings.index.lifecycle.name in template to {metricbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
        Sep 30 19:40:31 user metricbeat[40005]: 2020-09-30T19:40:31.363-0500#011INFO#011template/load.go:89#011Template metricbeat-7.9.2 already exists and will not be overwritten.
        Sep 30 19:40:31 user metricbeat[40005]: 2020-09-30T19:40:31.363-0500#011INFO#011[index-management]#011idxmgmt/std.go:298#011Loaded index template.
        Sep 30 19:40:31 user metricbeat[40005]: 2020-09-30T19:40:31.449-0500#011INFO#011[index-management]#011idxmgmt/std.go:309#011Write alias successfully generated.
        Sep 30 19:40:31 user metricbeat[40005]: 2020-09-30T19:40:31.536-0500#011INFO#011[publisher_pipeline_output]#011pipeline/output.go:151#011Connection to backoff(elasticsearch(https://4a11cf6113d94bffbf45eabb85ab2adf.us-east-1.aws.found.io:443)) established
        Sep 30 19:40:39 user metricbeat[40005]: 2020-09-30T19:40:39.496-0500#011INFO#011[monitoring]#011log/log.go:145#011Non-zero metrics in the last 30s#011{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":170,"time":{"ms":180}},"total":{"ticks":610,"time":{"ms":625},"value":610},"user":{"ticks":440,"time":{"ms":445}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":16},"info":{"ephemeral_id":"f4cd3a9c-d512-4ab0-9a94-67d0b11c33cc","uptime":{"ms":30110}},"memstats":{"gc_next":18922080,"memory_alloc":14468592,"memory_total":46834600,"rss":82595840},"runtime":{"goroutines":42}},"libbeat":{"config":{"module":{"running":1,"starts":1},"reloads":1,"scans":1},"output":{"events":{"acked":13,"batches":1,"total":13},"type":"elasticsearch"},"pipeline":{"clients":1,"events":{"active":0,"published":13,"retry":13,"total":13},"queue":{"acked":13}}},"metricbeat":{"aws":{"cloudwatch":{"events":13,"success":13}}},"system":{"cpu":{"cores":8},"load":{"1":0.29,"15":0.78,"5":0.65,"norm":{"1":0.0363,"15":0.0975,"5":0.0813}}}}}}
#4

It looks like it has published something, per that last line.

What is the output from the _cat/indices?v endpoint?

#5

Sorry. I don't understand. You mean an elastic cluster endpoint or a file?

#6

Elasticsearch endpoint.

#7

Sorry haha.

health status index                              uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   apm-7.9.2-metric-000001            xxxxxxxxxxxxxxxxxxxxxx   1   1          0            0       416b           208b
green  open   .kibana-event-log-7.9.2-000001     xxxxxxxxxxxxxxxxxxxxxx   1   1          2            0     21.8kb         10.8kb
green  open   .apm-agent-configuration           xxxxxxxxxxxxxxxxxxxxxx   1   1          0            0       416b           208b
green  open   .kibana_1                          xxxxxxxxxxxxxxxxxxxxxx   1   1       1753          206     23.3mb         11.6mb
green  open   .security-tokens-7                 xxxxxxxxxxxxxxxxxxxxxx   1   1         18            1    234.9kb        111.4kb
green  open   .security-7                        xxxxxxxxxxxxxxxxxxxxxx   1   1         52            0    253.1kb        121.3kb
green  open   apm-7.9.2-profile-000001           xxxxxxxxxxxxxxxxxxxxxx   1   1          0            0       416b           208b
green  open   .apm-custom-link                   xxxxxxxxxxxxxxxxxxxxxx   1   1          0            0       416b           208b
green  open   .kibana_task_manager_1             lxxxxxxxxxxxxxxxxxxxxxx  1   1          6           14    377.8kb          173kb
green  open   apm-7.9.2-transaction-000001       xxxxxxxxxxxxxxxxxxxxxx   1   1          0            0       416b           208b
green  open   apm-7.9.2-onboarding-2020.09.30    xxxxxxxxxxxxxxxxxxxxxx   1   1          1            0     14.6kb          7.3kb
green  open   metricbeat-7.9.2-2020.09.30-000001 xxxxxxxxxxxxxxxxxxxxxx   1   1        754            0    763.6kb          371kb
green  open   apm-7.9.2-span-000001              xxxxxxxxxxxxxxxxxxxxxx   1   1          0            0       416b           208b
green  open   apm-7.9.2-error-000001             xxxxxxxxxxxxxxxxxxxxxx   1   1          0            0       416b           208b
#8

You've definitely got data in there then. Have you looked in Discover in Kibana?

1 Like
#9

Yes, you're right. There is data on the Discover Tab.

#10

Awesome, so does everything look ok now?