I'm working on a simple ingesting Cloudtrail logs into our AWS managed OpenSearch cluster.
For installation, I'm using an Ubuntu 22.04 ARM64 EC2 instance. My logstash installation script is fairly simple, downloading the OSS version of logstash and installing the opensearch output plugin
export LOGSTASH_VERSION="8.12.2"
curl -O https://artifacts.elastic.co/downloads/logstash/logstash-oss-${LOGSTASH_VERSION}-linux-aarch64.tar.gz
tar -zxvf logstash-oss-${LOGSTASH_VERSION}-linux-aarch64.tar.gz
cd logstash-${LOGSTASH_VERSION}
./bin/logstash-plugin install logstash-output-opensearch
My logstash conf file is:
input {
s3 {
bucket => "<bucket-name>"
delete => true
interval => 60 # seconds
prefix => "cloudtrail/"
type => "cloudtrail"
codec => "cloudtrail" { charset => "UTF-8" }
region => "us-west-2"
backup_to_bucket => "<bucket-name>"
backup_add_prefix => "processed-logs/"
}
}
output {
if [type] == "cloudtrail" {
opensearch {
hosts => ["<host>"]
http_compression => true
index => "logstash-cloudtrail"
# Using IMDS STS Authentication
auth_type => {
type => "aws_iam"
region => "us-west-2"
service_name => "es"
}
}
}
}
When I attempt to run this configuration I get an error that the cloudtrail codec plugin is not installed
[2024-03-21T18:07:13,103][ERROR][logstash.plugins.registry] Unable to load plugin. {:type=>"codec", :name=>"\"cloudtrail\""}
[2024-03-21T18:07:13,112][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (PluginLoadingError) Couldn't find any codec plugin named '\"cloudtrail\"'. Are you sure this is correct? Trying to load the \"cloudtrail\" codec plugin resulted in this error: Unable to load the requested plugin named \"cloudtrail\" of type codec. The plugin is not installed."
However when I list the installed plugins, the cloudtrail codec is listed as installed under the logstash-integration-aws plugin that comes preinstalled in the tar archive:
$ ./bin/logstash-plugin list
Using bundled JDK: /home/ubuntu/logstash-8.12.2/jdk
io/console on JRuby shells out to stty for most operations
logstash-codec-avro
logstash-codec-cef
logstash-codec-collectd
logstash-codec-dots
logstash-codec-edn
logstash-codec-edn_lines
logstash-codec-es_bulk
logstash-codec-fluent
logstash-codec-graphite
logstash-codec-json
logstash-codec-json_lines
logstash-codec-line
logstash-codec-msgpack
logstash-codec-multiline
logstash-codec-netflow
logstash-codec-plain
logstash-codec-rubydebug
logstash-filter-aggregate
logstash-filter-anonymize
logstash-filter-cidr
logstash-filter-clone
logstash-filter-csv
logstash-filter-date
logstash-filter-de_dot
logstash-filter-dissect
logstash-filter-dns
logstash-filter-drop
logstash-filter-elasticsearch
logstash-filter-fingerprint
logstash-filter-geoip
logstash-filter-grok
logstash-filter-http
logstash-filter-json
logstash-filter-kv
logstash-filter-memcached
logstash-filter-metrics
logstash-filter-mutate
logstash-filter-prune
logstash-filter-ruby
logstash-filter-sleep
logstash-filter-split
logstash-filter-syslog_pri
logstash-filter-throttle
logstash-filter-translate
logstash-filter-truncate
logstash-filter-urldecode
logstash-filter-useragent
logstash-filter-uuid
logstash-filter-xml
logstash-input-azure_event_hubs
logstash-input-beats
└── logstash-input-elastic_agent (alias)
logstash-input-couchdb_changes
logstash-input-dead_letter_queue
logstash-input-elastic_serverless_forwarder
logstash-input-elasticsearch
logstash-input-exec
logstash-input-file
logstash-input-ganglia
logstash-input-gelf
logstash-input-generator
logstash-input-graphite
logstash-input-heartbeat
logstash-input-http
logstash-input-http_poller
logstash-input-imap
logstash-input-jms
logstash-input-pipe
logstash-input-redis
logstash-input-snmp
logstash-input-snmptrap
logstash-input-stdin
logstash-input-syslog
logstash-input-tcp
logstash-input-twitter
logstash-input-udp
logstash-input-unix
logstash-integration-aws
├── logstash-codec-cloudfront
├── logstash-codec-cloudtrail # CODEC INSTALLED HERE
├── logstash-input-cloudwatch
├── logstash-input-s3
├── logstash-input-sqs
├── logstash-output-cloudwatch
├── logstash-output-s3
├── logstash-output-sns
└── logstash-output-sqs
logstash-integration-elastic_enterprise_search
├── logstash-output-elastic_app_search
└── logstash-output-elastic_workplace_search
logstash-integration-jdbc
├── logstash-input-jdbc
├── logstash-filter-jdbc_streaming
└── logstash-filter-jdbc_static
logstash-integration-kafka
├── logstash-input-kafka
└── logstash-output-kafka
logstash-integration-logstash
├── logstash-input-logstash
└── logstash-output-logstash
logstash-integration-rabbitmq
├── logstash-input-rabbitmq
└── logstash-output-rabbitmq
logstash-output-csv
logstash-output-elasticsearch
logstash-output-email
logstash-output-file
logstash-output-graphite
logstash-output-http
logstash-output-lumberjack
logstash-output-nagios
logstash-output-null
logstash-output-opensearch
logstash-output-pipe
logstash-output-redis
logstash-output-stdout
logstash-output-tcp
logstash-output-udp
logstash-output-webhdfs
logstash-patterns-core
Full error output:
$ /home/ubuntu/logstash-8.12.2/bin/logstash -f /home/ubuntu/logstash-8.12.2/config/logstash.conf
Using bundled JDK: /home/ubuntu/logstash-8.12.2/jdk
/home/ubuntu/logstash-8.12.2/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_int
/home/ubuntu/logstash-8.12.2/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_f
Sending Logstash logs to /home/ubuntu/logstash-8.12.2/logs which is now configured via log4j2.properties
[2024-03-21T18:07:11,643][INFO ][logstash.runner ] Log4j configuration path used is: /home/ubuntu/logstash-8.12.2/config/log4j2.properties
[2024-03-21T18:07:11,649][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.12.2", "jruby.version"=>"jruby 9.4.5.0 (3.1.4) 2023-11-02 1abae2700f OpenJDK 64-Bit Server VM 17.0.10+7 on 17.0.10+7 +indy +jit [aarch64-linux]"}
[2024-03-21T18:07:11,655][INFO ][logstash.runner ] JVM bootstrap flags: [-XX:+HeapDumpOnOutOfMemoryError, -Dlogstash.jackson.stream-read-constraints.max-number-length=10000, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, -Djruby.regexp.interruptible=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11, -Dlog4j2.isThreadContextMapInheritable=true, -Xms1g, -Dlogstash.jackson.stream-read-constraints.max-string-length=200000000, -Djdk.io.File.enableADS=true, -Dfile.encoding=UTF-8, --add-opens=java.base/java.io=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, -Djruby.compile.invokedynamic=true, -Xmx1g, -Djava.security.egd=file:/dev/urandom, -Djava.awt.headless=true, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED]
[2024-03-21T18:07:11,658][INFO ][logstash.runner ] Jackson default value override `logstash.jackson.stream-read-constraints.max-string-length` configured to `200000000`
[2024-03-21T18:07:11,659][INFO ][logstash.runner ] Jackson default value override `logstash.jackson.stream-read-constraints.max-number-length` configured to `10000`
[2024-03-21T18:07:11,671][INFO ][logstash.settings ] Creating directory {:setting=>"path.queue", :path=>"/home/ubuntu/logstash-8.12.2/data/queue"}
[2024-03-21T18:07:11,675][INFO ][logstash.settings ] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/home/ubuntu/logstash-8.12.2/data/dead_letter_queue"}
[2024-03-21T18:07:11,864][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2024-03-21T18:07:11,876][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"ad32e00d-c676-4d6a-8725-8ca5e3a30032", :path=>"/home/ubuntu/logstash-8.12.2/data/uuid"}
[2024-03-21T18:07:12,438][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2024-03-21T18:07:13,052][INFO ][org.reflections.Reflections] Reflections took 155 ms to scan 1 urls, producing 132 keys and 468 values
[2024-03-21T18:07:13,103][ERROR][logstash.plugins.registry] Unable to load plugin. {:type=>"codec", :name=>"\"cloudtrail\""}
[2024-03-21T18:07:13,112][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (PluginLoadingError) Couldn't find any codec plugin named '\"cloudtrail\"'. Are you sure this is correct? Trying to load the \"cloudtrail\" codec plugin resulted in this error: Unable to load the requested plugin named \"cloudtrail\" of type codec. The plugin is not installed.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:186)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:847)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1319)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:139)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:446)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.RubyClass.newInstance(RubyClass.java:931)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:446)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:548)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:88)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:228)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:476)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:293)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:328)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:144)", "org.jruby.RubyProc.call(RubyProc.java:352)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111)", "java.base/java.lang.Thread.run(Thread.java:840)"]}
[2024-03-21T18:07:13,131][INFO ][logstash.runner ] Logstash shut down.
[2024-03-21T18:07:13,136][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:808) ~[jruby.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:767) ~[jruby.jar:?]
at home.ubuntu.logstash_minus_8_dot_12_dot_2.lib.bootstrap.environment.<main>(/home/ubuntu/logstash-8.12.2/lib/bootstrap/environment.rb:90) ~[?:?]
I double checked the file permissions, ran logstash as root just in case, uninstalled the logstash-integration-aws plugin and installed just the logstash-codec-cloudtrail by itself. All resulting in the same error of logstash not finding the installed codec plugin.
What am I missing here?
)