Logstash Inplut plugin metrics and filters?

Elastic Stack Logstash
1

image
image.png776×741 46.3 KB

From the image above, we can see the metrics and filters for variety of namesspaces including EC2 and EBS, but how about logs? I don't know what to enter for metrics and filters if I want to import AWS Cloudwatch logs to Logstsah inputp plugin.

My code so far:

 input{
  cloudwatch {
    namespace => "AWS/Logs"
    metrics => ["IncomingLogEvents"] # ?
    filters => {"logStream:/aws/codebuild/b-c-api" => "d3dc98f5-2c04-4416-9297-9051d8e2a2fb"}    # ?
    # log_group => "/aws/codebuild/b-c-api"
    access_key_id => "...id..."
    secret_access_key => "...secret..."
    region => "us-west-2"
    # type => "lambda"
  }
}

output{
  stdout{
codec => rubydebug  # outputs to command line
  }
}

output:

~/Documents/logstash-7.0.1$ ./bin/logstash -f config/pipelines/cloudwatchPipeline.conf 
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/home/kourosh/Documents/logstash-7.0.1/logstash-core/lib/jars/jruby-complete-9.2.7.0.jar) to field java.io.FileDescriptor.fd
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Sending Logstash logs to /home/kourosh/Documents/logstash-7.0.1/logs which is now configured via log4j2.properties
[2019-05-24T11:00:59,711][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-05-24T11:00:59,719][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.0.1"}
[2019-05-24T11:01:01,515][INFO ][logstash.javapipeline    ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, :thread=>"#<Thread:0x751a4b62 run>"}
[2019-05-24T11:01:01,570][INFO ][logstash.javapipeline    ] Pipeline started {"pipeline.id"=>"main"}
[2019-05-24T11:01:01,614][INFO ][logstash.inputs.cloudwatch] Polling CloudWatch API
[2019-05-24T11:01:01,668][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-05-24T11:01:01,974][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9601}
[2019-05-24T11:01:04,279][INFO ][logstash.inputs.cloudwatch] [Aws::CloudWatch::Client 200 1.420036 0 retries] list_metrics(namespace:"AWS/Logs")  

[2019-05-24T11:01:04,462][INFO ][logstash.inputs.cloudwatch] [Aws::CloudWatch::Client 200 0.094533 0 retries] get_metric_statistics(namespace:"AWS/Logs",metric_name:"IncomingLogEvents",start_time:2019-05-24 17:46:04 UTC,end_time:2019-05-24 18:01:04 UTC,period:300,statistics:["SampleCount","Average","Minimum","Maximum","Sum"],dimensions:[{name:"logStream:/aws/codebuild/b-c-api",value:"[FILTERED]"}])

I don't get any logs showing up in my terminal output even though it says successfully started Logstash API endpoint...

1 Like
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.