From the image above, we can see the metrics and filters for variety of namesspaces including EC2 and EBS, but how about logs? I don't know what to enter for metrics and filters if I want to import AWS Cloudwatch logs to Logstsah inputp plugin.
My code so far:
input{
cloudwatch {
namespace => "AWS/Logs"
metrics => ["IncomingLogEvents"] # ?
filters => {"logStream:/aws/codebuild/b-c-api" => "d3dc98f5-2c04-4416-9297-9051d8e2a2fb"} # ?
# log_group => "/aws/codebuild/b-c-api"
access_key_id => "...id..."
secret_access_key => "...secret..."
region => "us-west-2"
# type => "lambda"
}
}
output{
stdout{
codec => rubydebug # outputs to command line
}
}
output:
~/Documents/logstash-7.0.1$ ./bin/logstash -f config/pipelines/cloudwatchPipeline.conf
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/home/kourosh/Documents/logstash-7.0.1/logstash-core/lib/jars/jruby-complete-9.2.7.0.jar) to field java.io.FileDescriptor.fd
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Sending Logstash logs to /home/kourosh/Documents/logstash-7.0.1/logs which is now configured via log4j2.properties
[2019-05-24T11:00:59,711][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-05-24T11:00:59,719][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.0.1"}
[2019-05-24T11:01:01,515][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, :thread=>"#<Thread:0x751a4b62 run>"}
[2019-05-24T11:01:01,570][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"main"}
[2019-05-24T11:01:01,614][INFO ][logstash.inputs.cloudwatch] Polling CloudWatch API
[2019-05-24T11:01:01,668][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-05-24T11:01:01,974][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9601}
[2019-05-24T11:01:04,279][INFO ][logstash.inputs.cloudwatch] [Aws::CloudWatch::Client 200 1.420036 0 retries] list_metrics(namespace:"AWS/Logs")
[2019-05-24T11:01:04,462][INFO ][logstash.inputs.cloudwatch] [Aws::CloudWatch::Client 200 0.094533 0 retries] get_metric_statistics(namespace:"AWS/Logs",metric_name:"IncomingLogEvents",start_time:2019-05-24 17:46:04 UTC,end_time:2019-05-24 18:01:04 UTC,period:300,statistics:["SampleCount","Average","Minimum","Maximum","Sum"],dimensions:[{name:"logStream:/aws/codebuild/b-c-api",value:"[FILTERED]"}])
I don't get any logs showing up in my terminal output even though it says successfully started Logstash API endpoint...