# Logstash Plugin for Cloudwatch logs

**URL:** <https://discuss.elastic.co/t/logstash-plugin-for-cloudwatch-logs/125677>\
**Category:** Logstash\
**Created:** [March 26, 2018, 8:54pm UTC](https://discuss.elastic.co/t/logstash-plugin-for-cloudwatch-logs/125677 "2018-03-26T20:54:05Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![krishna\_chaitanya](https://avatars.discourse-cdn.com/v4/letter/k/b5a626/32.png) [@krishna\_chaitanya](https://discuss.elastic.co/u/krishna_chaitanya)\
**Post date:** [March 26, 2018, 8:54pm UTC](https://discuss.elastic.co/t/logstash-plugin-for-cloudwatch-logs/125677/1 "2018-03-26T20:54:05Z")

</div>

Hi, I am trying to integrate AWS Lambda logs onto ELK Stack.  
My AWS Lambda is writing logs into Cloudwatch logs.

I am trying to get this data into logstash using [logstash-input-cloudwatch](https://www.elastic.co/guide/en/logstash/5.5/plugins-inputs-cloudwatch.html) plugin. But, having difficulties.

Below is my config.

```
input {
   cloudwatch {
       type => "cloudwatch_lambda"
       namespace => "AWS/Logs"
       filters => { "logStream:Group" => "MyLambdaStreamName" }
       region => "us-east-1"
    }
}

```

Just sending output to a file to see if I can ingest them or not.  
Errors in logstash log file:

```
[2018-03-26T16:16:55,008][INFO][logstash.inputs.cloudwatch] Polling CloudWatch API
[2018-03-26T16:16:55,009][ERROR][logstash.pipeline] A plugin had an unrecoverable error. Will restart this plugin.
  Plugin: <LogStash::Inputs::CloudWatch type=>"cloudwatch_lambda", namespace=>"AWS/Logs", filters=>{"logStream:Group"=>"MyLambdaStreamName"}, region=>"us-east-1", id=>"8716239dasdaddasskdbasjdldj-8", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_8763183n-dwqdka-dkdnka-dsakdna", enable_metric=>true, charset=>"UTF-8">, use_ssl=>true, metrics=>["CPUUtilization", "DiskReadOps", "DiskWriteOps", "NetworkIn", "NetworkOut"], statistics=>["SampleCount", "Average", "Minimum", "Maximum", "Sum"], interval=>900, period=>300, combined=>false>
  Error: No metrics to query
  Exception: RuntimeError
  Stack: /logstash-5.5.0/vendor/bundle/jruby/1.9/gems/logstash-input-cloudwatch-2.0.3/lib/logstash/inputs/cloudwatch.rb:144:in `run'
org/jruby/RubyProc.java:281:in `call'
/logstash-5.5.0-bundle/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/interval.rb:20:in `interval'
/logstash-5.5.0-bundle/vendor/bundle/jruby/1.9/gems/logstash-input-cloudwatch-2.0.3/lib/logstash/inputs/cloudwatch.rb:141:in `run'
/logstash-5.5.0-bundle/logstash-core/lib/logstash/pipeline.rb:456:in `inputworker'
/logstash-5.5.0-bundle/logstash-core/lib/logstash/pipeline.rb:449:in `start_input'

```

These logs are appearing every second.

I found this article: plugin developed specifically to ingest cloudwatch logs: [`https://lukewaite.ca/aws/lambda/elk/logstash/2015/07/13/aws-lambda-and-elk.html`](https://lukewaite.ca/aws/lambda/elk/logstash/2015/07/13/aws-lambda-and-elk.html)  
I will try this, but this is not in the list of official plugins, so if possible, I would like to do it using official Clouwatch plugin for logstash.

Logstash version 5.5.0. Any help appreciated.

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [March 27, 2018, 12:35am UTC](https://discuss.elastic.co/t/logstash-plugin-for-cloudwatch-logs/125677/2 "2018-03-27T00:35:01Z")

</div>

It looks like you haven't specified any metrics that are compatible with the selected namespace.

> `metrics`
> 
> - Value type is array
> - Default value is `["CPUUtilization", "DiskReadOps", "DiskWriteOps", "NetworkIn", "NetworkOut"]`
> 
> Specify the metrics to fetch for the namespace. The defaults are AWS/EC2 specific. See [AWS services that publish CloudWatch metrics - Amazon CloudWatch](http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/aws-namespaces.html) for the available metrics for other namespaces.
> 
> -- [Elastic Docs, Cloudwatch Input, metrics parameter](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-cloudwatch.html#plugins-inputs-cloudwatch-metrics)

The metrics available for your selected namespace `AWS/Logs` can be found in the [`Amazon CloudWatch Logs`](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cwl-metricscollected.html) sub-page.

---

<div class="post-metadata">

**Author:** ![krishna\_chaitanya](https://avatars.discourse-cdn.com/v4/letter/k/b5a626/32.png) [@krishna\_chaitanya](https://discuss.elastic.co/u/krishna_chaitanya)\
**Post date:** [March 27, 2018, 1:03am UTC](https://discuss.elastic.co/t/logstash-plugin-for-cloudwatch-logs/125677/3 "2018-03-27T01:03:22Z")

</div>

I skipped the metrics.  
Since I am interested in AWS Lambda logs which are being written to Cloudwatch logs,  
I require Cloudwatch logs entirely. Not the metrics.

---

<div class="post-metadata">

**Author:** ![krishna\_chaitanya](https://avatars.discourse-cdn.com/v4/letter/k/b5a626/32.png) [@krishna\_chaitanya](https://discuss.elastic.co/u/krishna_chaitanya)\
**Post date:** [March 29, 2018, 4:57pm UTC](https://discuss.elastic.co/t/logstash-plugin-for-cloudwatch-logs/125677/4 "2018-03-29T16:57:17Z")

</div>

Any help please?

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [March 29, 2018, 5:20pm UTC](https://discuss.elastic.co/t/logstash-plugin-for-cloudwatch-logs/125677/5 "2018-03-29T17:20:06Z")

</div>

The plugin you're using appears to only support _metrics_ from cloudwatch, which is why it is failing when it is configured in a manner that has no metrics.

There is a community-provided plugin [`lukewaite/logstash-input-cloudwatch-logs`](https://github.com/lukewaite/logstash-input-cloudwatch-logs) that claims to get the logs themselves from cloudwatch. The plugin does have a few open issues, but it may be the best place to get started.

---

<div class="post-metadata">

**Author:** ![krishna\_chaitanya](https://avatars.discourse-cdn.com/v4/letter/k/b5a626/32.png) [@krishna\_chaitanya](https://discuss.elastic.co/u/krishna_chaitanya)\
**Post date:** [March 29, 2018, 5:33pm UTC](https://discuss.elastic.co/t/logstash-plugin-for-cloudwatch-logs/125677/6 "2018-03-29T17:33:03Z")

</div>

Thanks so much. Will try this out

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 26, 2018, 5:33pm UTC](https://discuss.elastic.co/t/logstash-plugin-for-cloudwatch-logs/125677/7 "2018-04-26T17:33:10Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.