Logstash S3 output plugin fails to upload txt files in S3

Elastic Stack Logstash
1

Hi,
We have tried to upload a txt file through logstash s3 output plugin but having issues it says:
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console

My logstash.conf config is like:

input {
  file {
    path => ["/var/log/test/abc.txt"]
    sincedb_path => "/dev/null"
    start_position => "beginning"
  }
}

filter {
  # Your filters if needed
}

output {
  s3 {
    access_key_id => "ID"
    secret_access_key => "key"
    region => "us-east-1"
    bucket => "qa-alb-logs"
    #size_file => 1
    time_file => 1
    prefix => "test/"
  }
}

need to get a step by step guide for this.

2

Have you tried reinstalling Logstash service?

3

yes, freshly installed on another VM, but the same issue persists.

1 Like
4

What are the errors? You didn't share any log, without it is not possible to know what may be happening.

5

Hi leandrojmp,

Here are the logs:

tail -f /var/log/logstash/logstash-plain.log.log
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]
[2024-02-15T10:05:06,590][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2024-02-15T10:05:06,599][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.17.18", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.20+8 on 11.0.20+8 +indy +jit [linux-aarch64]"}
[2024-02-15T10:05:06,601][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true]
[2024-02-15T10:05:07,830][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2024-02-15T10:05:07,833][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/etc/logstash/conf.d/*.conf"}
[2024-02-15T10:05:07,842][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.

and when we running it the console output is:

/usr/share/logstash/bin/logstash -f logstash.conf
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2024-02-15 15:39:00.748 [main] runner - Starting Logstash {"logstash.version"=>"7.17.18", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.20+8 on 11.0.20+8 +indy +jit [linux-aarch64]"}
[INFO ] 2024-02-15 15:39:00.754 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true]
[WARN ] 2024-02-15 15:39:01.006 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2024-02-15 15:39:01.976 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9601, :ssl_enabled=>false}
[INFO ] 2024-02-15 15:39:03.067 [Converge PipelineAction::Create<main>] Reflections - Reflections took 73 ms to scan 1 urls, producing 119 keys and 419 values 
[WARN ] 2024-02-15 15:39:03.869 [Converge PipelineAction::Create<main>] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2024-02-15 15:39:03.920 [Converge PipelineAction::Create<main>] file - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2024-02-15 15:39:14.056 [Converge PipelineAction::Create<main>] line - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2024-02-15 15:39:35.926 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/root/logstash.conf"], :thread=>"#<Thread:0x40381236 run>"}
[INFO ] 2024-02-15 15:39:36.558 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>0.63}
[INFO ] 2024-02-15 15:39:36.610 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2024-02-15 15:39:36.649 [[main]<file] observingtail - START, creating Discoverer, Watch with file and sincedb collections
[INFO ] 2024-02-15 15:39:36.672 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}

Thanks

6

Can you please share the ls -al for the etc/logstash and ./conf.d
Just want to know if the file listed under is config or conf, as *.conf is the regex.

Also I found the similar ticket resolved: Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties

If you are not version specific try installing new version for the Logstash

7

Most likely you have permission issue. The "logstash" user which run the service, doesn't have rights to read conf directory or files.

If I'm not wrong, your .conf file is in the /usr/share/logstash/bin/ directory. It should be in /etc/logstash/conf.d

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.