# CloudWatch Integration to ELK Stack

**URL:** https://discuss.elastic.co/t/cloudwatch-integration-to-elk-stack/163064
**Category:** Logstash
**Created:** [January 6, 2019, 5:29am UTC](https://discuss.elastic.co/t/cloudwatch-integration-to-elk-stack/163064 "2019-01-06T05:29:29Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![Mayank\_Khudia](https://avatars.discourse-cdn.com/v4/letter/m/7ab992/32.png) [@Mayank\_Khudia](https://discuss.elastic.co/u/Mayank_Khudia)
#### Post date: [January 6, 2019, 5:29am UTC](https://discuss.elastic.co/t/cloudwatch-integration-to-elk-stack/163064/1 "2019-01-06T05:29:29Z")

</div>

Hello Team,  
I am trying to integrate CloudWatch Logs to ELK stack. I have successfully managed to install all 3 components of the stack. But still unable to monitor and visualize log streams from CloudWatch. I have set up the ELK stack on AWS t2.large EC2 instance in us-east-1 region.

Following are my config files for Logstash/ElasticSearch/Kibana respectively. Please help:  
_ **Logstash.yml** _

# Settings file in YAML

# Settings can be specified either in hierarchical form, e.g.:

# pipeline:

# batch:

# size: 125

# delay: 5

# Or as flat keys:

# pipeline.batch.size: 125

# pipeline.batch.delay: 5

# ------------ Node identity ------------

# Use a descriptive name for the node:

# node.name: test

# If omitted the node name will default to the machine's host name

# ------------ Data path ------------------

# Which directory should be used by logstash and its plugins

# for any persistent needs. Defaults to LOGSTASH\_HOME/data

path.data: /var/lib/logstash

# ------------ Pipeline Settings --------------

# The ID of the pipeline.

# pipeline.id: main

# Set the number of workers that will, in parallel, execute the filters+outputs

# stage of the pipeline.

# This defaults to the number of the host's CPU cores.

# pipeline.workers: 2

# How many events to retrieve from inputs before sending to filters+workers

# pipeline.batch.size: 125

# How long to wait in milliseconds while polling for the next event

# before dispatching an undersized batch to filters+outputs

# pipeline.batch.delay: 50

# Force Logstash to exit during shutdown even if there are still inflight

# events in memory. By default, logstash will refuse to quit until all

# received events have been pushed to the outputs.

# WARNING: enabling this can lead to data loss during shutdown

# pipeline.unsafe\_shutdown: false

# ------------ Pipeline Configuration Settings --------------

# Where to fetch the pipeline configuration for the main pipeline

# path.config:

# Pipeline configuration string for the main pipeline

# config.string:

# At startup, test if the configuration is valid and exit (dry run)

# config.test\_and\_exit: false

# Periodically check if the configuration has changed and reload the pipeline

# This can also be triggered manually through the SIGHUP signal

# config.reload.automatic: false

#How often to check if the pipeline configuration has changed (in seconds)

# config.reload.interval: 3s

# Show fully compiled configuration as debug log message

# NOTE: --log.level must be 'debug'

# config.debug: false

# When enabled, process escaped characters such as \n and " in strings in the

# pipeline configuration files.

# config.support\_escapes: false

# ------------ Module Settings ---------------

# Define modules here. Modules definitions must be defined as an array.

# The simple way to see this is to prepend each `name` with a `-`, and keep

# all associated variables under the `name` they are associated with, and

# above the next, like this:

# modules:

# - name: MODULE\_NAME

# var.PLUGINTYPE1.PLUGINNAME1.KEY1: VALUE

# var.PLUGINTYPE1.PLUGINNAME1.KEY2: VALUE

# var.PLUGINTYPE2.PLUGINNAME1.KEY1: VALUE

# var.PLUGINTYPE3.PLUGINNAME3.KEY1: VALUE

# Module variable names must be in the format of

# var.PLUGIN\_TYPE.PLUGIN\_NAME.KEY

# modules:

# ------------ Cloud Settings ---------------

# Define Elastic Cloud settings here.

# Format of cloud.id is a base64 value e.g. dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRub3RhcmVhbCRpZGVudGlmaWVy

# and it may have an label prefix e.g. staging:dXMtZ...

# This will overwrite 'var.elasticsearch.hosts' and 'var.kibana.host'

# cloud.id: \<identifier\>

# Format of cloud.auth is: \<user\>:\<pass\>

# This is optional

# If supplied this will overwrite 'var.elasticsearch.username' and 'var.elasticsearch.password'

# If supplied this will overwrite 'var.kibana.username' and 'var.kibana.password'

# cloud.auth: elastic:\<password\>

# ------------ Queuing Settings --------------

# Internal queuing model, "memory" for legacy in-memory based queuing and

# "persisted" for disk-based acked queueing. Defaults is memory

# queue.type: memory

# If using queue.type: persisted, the directory path where the data files will be stored.

# Default is path.data/queue

# path.queue:

# If using queue.type: persisted, the page data files size. The queue data consists of

# append-only data files separated into pages. Default is 64mb

# queue.page\_capacity: 64mb

# If using queue.type: persisted, the maximum number of unread events in the queue.

# Default is 0 (unlimited)

# queue.max\_events: 0

# If using queue.type: persisted, the total capacity of the queue in number of bytes.

# If you would like more unacked events to be buffered in Logstash, you can increase the

# capacity using this setting. Please make sure your disk drive has capacity greater than

# the size specified here. If both max\_bytes and max\_events are specified, Logstash will pick

# whichever criteria is reached first

# Default is 1024mb or 1gb

# queue.max\_bytes: 1024mb

# If using queue.type: persisted, the maximum number of acked events before forcing a checkpoint

# Default is 1024, 0 for unlimited

# queue.checkpoint.acks: 1024

# If using queue.type: persisted, the maximum number of written events before forcing a checkpoint

# Default is 1024, 0 for unlimited

# queue.checkpoint.writes: 1024

# If using queue.type: persisted, the interval in milliseconds when a checkpoint is forced on the head page

# Default is 1000, 0 for no periodic checkpoint.

# queue.checkpoint.interval: 1000

# ------------ Dead-Letter Queue Settings --------------

# Flag to turn on dead-letter queue.

# dead\_letter\_queue.enable: false

# If using dead\_letter\_queue.enable: true, the maximum size of each dead letter queue. Entries

# will be dropped if they would increase the size of the dead letter queue beyond this setting.

# Default is 1024mb

# dead\_letter\_queue.max\_bytes: 1024mb

# If using dead\_letter\_queue.enable: true, the directory path where the data files will be stored.

# Default is path.data/dead\_letter\_queue

# path.dead\_letter\_queue:

**# ------------ Metrics Settings --------------**  
**# Bind address for the metrics REST endpoint**  
**# http.host: "127.0.0.1"**  
**# Bind port for the metrics REST endpoint, this option also accept a range**  
**# (9600-9700) and logstash will pick up the first available ports.**  
**# http.port: 9600-9700**

# ------------ Debugging Settings --------------

# Options for log.level:

# \* fatal

# \* error

# \* warn

# \* info (default)

# \* debug

# \* trace

# log.level: info

path.logs: /var/log/logstash

# ------------ Other Settings --------------

# Where to find custom plugins

# path.plugins:

---

<div class="post-metadata">

### Author: ![Mayank\_Khudia](https://avatars.discourse-cdn.com/v4/letter/m/7ab992/32.png) [@Mayank\_Khudia](https://discuss.elastic.co/u/Mayank_Khudia)
#### Post date: [January 6, 2019, 5:32am UTC](https://discuss.elastic.co/t/cloudwatch-integration-to-elk-stack/163064/2 "2019-01-06T05:32:10Z")

</div>

Below is Logstash conf file under /etc/Logstash/conf.d/

input {

cloudwatch {

namespace =\> "AWS/EC2"

metrics =\> ["CPUUtilization"]

filters =\> { "tag:Group" =\> "Kibana" }

region =\> "us-east-1"

port =\>

}

}

output {

elasticsearch { hosts =\> ["localhost:9200"] }

stdout { codec =\> rubydebug }

}

---

<div class="post-metadata">

### Author: ![Mayank\_Khudia](https://avatars.discourse-cdn.com/v4/letter/m/7ab992/32.png) [@Mayank\_Khudia](https://discuss.elastic.co/u/Mayank_Khudia)
#### Post date: [January 6, 2019, 5:33am UTC](https://discuss.elastic.co/t/cloudwatch-integration-to-elk-stack/163064/3 "2019-01-06T05:33:15Z")

</div>

My ElasticSearch seems to be working giving me a response like:  
{  
"name" : "XYn\_Lrk",  
"cluster\_name" : "elasticsearch",  
"cluster\_uuid" : "D0lCfrNrTF62cNIVA1xlGA",  
"version" : {  
"number" : "6.2.4",  
"build\_hash" : "ccec39f",  
"build\_date" : "2018-04-12T20:37:28.497551Z",  
"build\_snapshot" : false,  
"lucene\_version" : "7.2.1",  
"minimum\_wire\_compatibility\_version" : "5.6.0",  
"minimum\_index\_compatibility\_version" : "5.0.0"  
},  
"tagline" : "You Know, for Search"  
}

---

<div class="post-metadata">

### Author: ![Mayank\_Khudia](https://avatars.discourse-cdn.com/v4/letter/m/7ab992/32.png) [@Mayank\_Khudia](https://discuss.elastic.co/u/Mayank_Khudia)
#### Post date: [January 6, 2019, 5:39am UTC](https://discuss.elastic.co/t/cloudwatch-integration-to-elk-stack/163064/4 "2019-01-06T05:39:26Z")

</div>

Below is my kibana.yml file:

Kibana is served by a back end server. This setting specifies the port to use.

server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.

# The default is 'localhost', which usually means remote machines will not be able to connect.

# To allow connections from remote users, set this parameter to a non-loopback address.

server.host: "localhost"

# Enables you to specify a path to mount Kibana at if you are running behind a proxy.

# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath

# from requests it receives, and to prevent a deprecation warning at startup.

# This setting cannot end in a slash.

#server.basePath: ""

# Specifies whether Kibana should rewrite requests that are prefixed with

# `server.basePath` or require that they are rewritten by your reverse proxy.

# This setting was effectively always `false` before Kibana 6.3 and will

# default to `true` starting in Kibana 7.0.

#server.rewriteBasePath: false

# The maximum payload size in bytes for incoming server requests.

#server.maxPayloadBytes: 1048576

# The Kibana server's name. This is used for display purposes.

server.name: "KibanaDashboard"

# The URL of the Elasticsearch instance to use for all your queries.

elasticsearch.url: "[http://localhost:9200](http://localhost:9200)"

# When this setting's value is true Kibana uses the hostname specified in the server.host

# setting. When the value of this setting is false, Kibana uses the hostname of the host

# that connects to this Kibana instance.

#elasticsearch.preserveHost: true

# Kibana uses an index in Elasticsearch to store saved searches, visualizations and

# dashboards. Kibana creates a new index if the index doesn't already exist.

#kibana.index: ".kibana"

# The default application to load.

#kibana.defaultAppId: "home"

# If your Elasticsearch is protected with basic authentication, these settings provide

# the username and password that the Kibana server uses to perform maintenance on the Kibana

# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which

# is proxied through the Kibana server.

#elasticsearch.username: "user"

#elasticsearch.password: "pass"

# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.

# These settings enable SSL for outgoing requests from the Kibana server to the browser.

#server.ssl.enabled: false  
server.ssl.certificate: /path/to/your/server.crt

#server.ssl.key: /path/to/your/server.key

# Optional settings that provide the paths to the PEM-format SSL certificate and key files.

# These files validate that your Elasticsearch backend uses the same key files.

#elasticsearch.ssl.certificate: /path/to/your/client.crt

#elasticsearch.ssl.key: /path/to/your/client.key

# Optional setting that enables you to specify a path to the PEM file for the certificate

# authority for your Elasticsearch instance.

#elasticsearch.ssl.certificateAuthorities: ["/path/to/your/CA.pem"]

# To disregard the validity of SSL certificates, change this setting's value to 'none'.

#elasticsearch.ssl.verificationMode: full

# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of

# the elasticsearch.requestTimeout setting.

#elasticsearch.pingTimeout: 1500

# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value

# must be a positive integer.

#elasticsearch.requestTimeout: 30000

# List of Kibana client-side headers to send to Elasticsearch. To send _no_ client-side

# headers, set this value to (an empty list).

#elasticsearch.requestHeadersWhitelist: [authorization]

# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten

# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.

#elasticsearch.customHeaders: {}

# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.

#elasticsearch.shardTimeout: 30000

# Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying.

#elasticsearch.startupTimeout: 5000

# Logs queries sent to Elasticsearch. Requires logging.verbose set to true.

#elasticsearch.logQueries: false

# Specifies the path where Kibana creates the process ID file.

#pid.file: /var/run/kibana.pid

# Enables you specify a file where Kibana stores log output.

#logging.dest: stdout

# Set the value of this setting to true to suppress all logging output.

#logging.silent: false

# Set the value of this setting to true to suppress all logging output other than error messages.

#logging.quiet: false

# Set the value of this setting to true to log all events, including system usage information

# and all requests.

#logging.verbose: false

# Set the interval in milliseconds to sample system and process performance

# metrics. Minimum is 100ms. Defaults to 5000.

#ops.interval: 5000

# The default locale. This locale can be used in certain circumstances to substitute any missing

# translations.

#i18n.defaultLocale: "en"

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 3, 2019, 5:39am UTC](https://discuss.elastic.co/t/cloudwatch-integration-to-elk-stack/163064/5 "2019-02-03T05:39:39Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.