Logstash Input Plugin fails

Elastic Stack Logstash
1

Hi everyone, I'm trying to export AWS Cloudwatch logs to Logstash input plugin. However, I get an error message saying "Failed to execute action...".

[pipeline.conf file]:
input{
cloudwatch {
namespace => "AWS/Logs"
metrics => [""] # ?
filters => {"logStream:myGroupName" => "myStreamName"} # ?
log_group => "myGroupName"
access_key_id => "...id..."
secret_access_key => "...secret..."
region => "us-west-2"
}
}

output{
stdout{
codec => rubydebug
}
}

[Error Message]:

./bin/logstash -f config/pipelines/cloudwatchPipeline.conf --config.reload.automatic
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/home/kourosh/Documents/logstash-7.0.1/logstash-core/lib/jars/jruby-complete-9.2.7.0.jar) to field java.io.FileDescriptor.fd
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Sending Logstash logs to /home/kourosh/Documents/logstash-7.0.1/logs which is now configured via log4j2.properties
[2019-05-24T09:04:29,375][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-05-24T09:04:29,383][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.0.1"}
[2019-05-24T09:04:30,856][ERROR][logstash.inputs.cloudwatch] Unknown setting 'log_group' for cloudwatch
[2019-05-24T09:04:30,862][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Something is wrong with your configuration.", :backtrace=>["/home/kourosh/Documents/logstash-7.0.1/logstash-core/lib/logstash/config/mixin.rb:86:in config_init'", "/home/kourosh/Documents/logstash-7.0.1/logstash-core/lib/logstash/inputs/base.rb:60:ininitialize'", "org/logstash/plugins/PluginFactoryExt.java:255:in plugin'", "org/logstash/plugins/PluginFactoryExt.java:117:inbuildInput'", "org/logstash/execution/JavaBasePipelineExt.java:50:in initialize'", "/home/kourosh/Documents/logstash-7.0.1/logstash-core/lib/logstash/java_pipeline.rb:23:ininitialize'", "/home/kourosh/Documents/logstash-7.0.1/logstash-core/lib/logstash/pipeline_action/create.rb:36:in execute'", "/home/kourosh/Documents/logstash-7.0.1/logstash-core/lib/logstash/agent.rb:325:inblock in converge_state'"]}
[2019-05-24T09:04:31,070][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9601}
[2019-05-24T09:04:34,035][ERROR][logstash.inputs.cloudwatch] Unknown setting 'log_group' for cloudwatch

2

That suggests you are running an unsupported version of Java.

Remove that option from the input.

3

@Badger, do you know what I need to include for the filter and metrics? All the resources online cover AWS/EC2, etc, except AWS Cloudwatch logs, I believe cloudwatch logs need different information in the Logstash input plugin.

4

The AWS documentation tells you what metrics and dimensions are available.

5

@Badger

I've tried it and it works, but I don't get any output logs in terminal or locahost to display.

my metrics and filters are:

metrics => ["IncomingLogEvents", "ForwardedLogEvents"] # ?
filters => {"LogGroupName" => "/aws/codebuild/b2-raw-processor"}    # ?

output:

~/Documents/logstash-7.0.1$ ./bin/logstash -f config/pipelines/cloudwatchPipeline.conf 
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/home/kourosh/Documents/logstash-7.0.1/logstash-core/lib/jars/jruby-complete-9.2.7.0.jar) to field java.io.FileDescriptor.fd
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Sending Logstash logs to /home/kourosh/Documents/logstash-7.0.1/logs which is now configured via log4j2.properties
[2019-05-24T12:01:17,450][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-05-24T12:01:17,459][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.0.1"}
[2019-05-24T12:01:19,547][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-05-24T12:01:19,714][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-05-24T12:01:19,761][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>7}
[2019-05-24T12:01:19,764][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2019-05-24T12:01:19,783][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-05-24T12:01:19,792][INFO ][logstash.outputs.elasticsearch] Using default mapping template
[2019-05-24T12:01:19,799][INFO ][logstash.javapipeline    ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, :thread=>"#<Thread:0x2964eeed run>"}
[2019-05-24T12:01:19,923][INFO ][logstash.javapipeline    ] Pipeline started {"pipeline.id"=>"main"}
[2019-05-24T12:01:19,971][INFO ][logstash.inputs.cloudwatch] Polling CloudWatch API
[2019-05-24T12:01:20,009][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2019-05-24T12:01:20,021][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-05-24T12:01:20,262][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9601}
[2019-05-24T12:01:22,885][INFO ][logstash.inputs.cloudwatch] [Aws::CloudWatch::Client 200 1.667485 0 retries] list_metrics(namespace:"AWS/Logs")  

[2019-05-24T12:01:23,107][INFO ][logstash.inputs.cloudwatch] [Aws::CloudWatch::Client 200 0.113511 0 retries] get_metric_statistics(namespace:"AWS/Logs",metric_name:"IncomingLogEvents",start_time:2019-05-24 18:46:22 UTC,end_time:2019-05-24 19:01:22 UTC,period:300,statistics:["SampleCount","Average","Minimum","Maximum","Sum"],dimensions:[{name:"LogGroupName",value:"[FILTERED]"}])
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.