Logstash Plugin for Cloudwatch logs

Elastic Stack Logstash
1

Hi, I am trying to integrate AWS Lambda logs onto ELK Stack.
My AWS Lambda is writing logs into Cloudwatch logs.

I am trying to get this data into logstash using logstash-input-cloudwatch plugin. But, having difficulties.

Below is my config.

input {
   cloudwatch {
       type => "cloudwatch_lambda"
       namespace => "AWS/Logs"
       filters => { "logStream:Group" => "MyLambdaStreamName" }
       region => "us-east-1"
    }
}

Just sending output to a file to see if I can ingest them or not.
Errors in logstash log file:

[2018-03-26T16:16:55,008][INFO ][logstash.inputs.cloudwatch] Polling CloudWatch API
[2018-03-26T16:16:55,009][ERROR][logstash.pipeline        ] A plugin had an unrecoverable error. Will restart this plugin.
  Plugin: <LogStash::Inputs::CloudWatch type=>"cloudwatch_lambda", namespace=>"AWS/Logs", filters=>{"logStream:Group"=>"MyLambdaStreamName"}, region=>"us-east-1", id=>"8716239dasdaddasskdbasjdldj-8", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_8763183n-dwqdka-dkdnka-dsakdna", enable_metric=>true, charset=>"UTF-8">, use_ssl=>true, metrics=>["CPUUtilization", "DiskReadOps", "DiskWriteOps", "NetworkIn", "NetworkOut"], statistics=>["SampleCount", "Average", "Minimum", "Maximum", "Sum"], interval=>900, period=>300, combined=>false>
  Error: No metrics to query
  Exception: RuntimeError
  Stack: /logstash-5.5.0/vendor/bundle/jruby/1.9/gems/logstash-input-cloudwatch-2.0.3/lib/logstash/inputs/cloudwatch.rb:144:in `run'
org/jruby/RubyProc.java:281:in `call'
/logstash-5.5.0-bundle/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/interval.rb:20:in `interval'
/logstash-5.5.0-bundle/vendor/bundle/jruby/1.9/gems/logstash-input-cloudwatch-2.0.3/lib/logstash/inputs/cloudwatch.rb:141:in `run'
/logstash-5.5.0-bundle/logstash-core/lib/logstash/pipeline.rb:456:in `inputworker'
/logstash-5.5.0-bundle/logstash-core/lib/logstash/pipeline.rb:449:in `start_input'

These logs are appearing every second.

I found this article: plugin developed specifically to ingest cloudwatch logs: https://lukewaite.ca/aws/lambda/elk/logstash/2015/07/13/aws-lambda-and-elk.html
I will try this, but this is not in the list of official plugins, so if possible, I would like to do it using official Clouwatch plugin for logstash.

Logstash version 5.5.0. Any help appreciated.

2

It looks like you haven't specified any metrics that are compatible with the selected namespace.

metrics

  • Value type is array
  • Default value is ["CPUUtilization", "DiskReadOps", "DiskWriteOps", "NetworkIn", "NetworkOut"]

Specify the metrics to fetch for the namespace. The defaults are AWS/EC2 specific. See AWS services that publish CloudWatch metrics - Amazon CloudWatch for the available metrics for other namespaces.

-- Elastic Docs, Cloudwatch Input, metrics parameter

The metrics available for your selected namespace AWS/Logs can be found in the Amazon CloudWatch Logs sub-page.

3

I skipped the metrics.
Since I am interested in AWS Lambda logs which are being written to Cloudwatch logs,
I require Cloudwatch logs entirely. Not the metrics.

1 Like
4

Any help please?

5

The plugin you're using appears to only support metrics from cloudwatch, which is why it is failing when it is configured in a manner that has no metrics.

There is a community-provided plugin lukewaite/logstash-input-cloudwatch-logs that claims to get the logs themselves from cloudwatch. The plugin does have a few open issues, but it may be the best place to get started.

6

Thanks so much. Will try this out

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.