No Logs from logstash docker

Elastic Stack Logstash
1

I call logstash from commandline /usr/share/logstash/bin/logstash --path.settings=/usr/share/logstash/config -f /usr/share/logstash/conf.d/

But no logs written

bash-4.4$ ls -ld logs1
drwxrwxrwx 2 logstash logstash 6 Sep 12 18:36 logs1

logstash logs on console prints

+ /usr/share/logstash/bin/logstash --debug --path.settings=/usr/share/logstash/config -f /usr/share/logstash/conf.d/ -w 2
Using bundled JDK: /usr/share/logstash/jdk
Sending Logstash logs to /usr/share/logstash/logs1 which is now configured via log4j2.properties
[2023-09-12T18:16:41,052][INFO ][logstash.runner          ] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties

logstash.yml

path.logs: /usr/share/logstash/logs1
log.format: plain

Please let me know what is missing here? I appreciate your help in advance.

2

Under which Linux account are you running LS?
logstash logstash - this is default when you run LS as a service
When you run LS as a process, it's a currently logged in user.

3

Thank you @Rios for looking into it and offering your help.

The container starts with the logstash user.

bash-4.4$ ps -aux | grep -i logstash
logstash     1  0.0  0.0  11924  2860 ?        Ss   Sep12   0:00 /bin/bash /opt/config/startup.sh
logstash    21  1.7  5.6 14852976 1800832 ?    Sl   Sep12   9:02 /usr/share/logstash/jdk/bin/java -Xms2G -Xmx8G -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -Dlog4j2.isThreadContextMapInheritable=true -Djruby.regexp.interruptible=true -Djdk.io.File.enableADS=true --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-opens=java.base/java.security=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.nio.channels=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.management/sun.management=ALL-UNNAMED -cp /usr/share/logstash/vendor/jruby/lib/jruby.jar:/usr/share/logstash/logstash-core/lib/jars/checker-qual-3.12.0.jar:/usr/share/logstash/logstash-core/lib/jars/commons-codec-1.15.jar:/usr/share/logstash/logstash-core/lib/jars/commons-compiler-3.1.0.jar:/usr/share/logstash/logstash-core/lib/jars/commons-logging-1.2.jar:/usr/share/logstash/logstash-core/lib/jars/error_prone_annotations-2.11.0.jar:/usr/share/logstash/logstash-core/lib/jars/failureaccess-1.0.1.jar:/usr/share/logstash/logstash-core/lib/jars/google-java-format-1.15.0.jar:/usr/share/logstash/logstash-core/lib/jars/guava-31.1-jre.jar:/usr/share/logstash/logstash-core/lib/jars/httpclient-4.5.13.jar:/usr/share/logstash/logstash-core/lib/jars/httpcore-4.4.14.jar:/usr/share/logstash/logstash-core/lib/jars/j2objc-annotations-1.3.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-annotations-2.14.1.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-core-2.14.1.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-databind-2.14.1.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-dataformat-cbor-2.14.1.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-dataformat-yaml-2.14.1.jar:/usr/share/logstash/logstash-core/lib/jars/janino-3.1.0.jar:/usr/share/logstash/logstash-core/lib/jars/javassist-3.29.0-GA.jar:/usr/share/logstash/logstash-core/lib/jars/jsr305-3.0.2.jar:/usr/share/logstash/logstash-core/lib/jars/jvm-options-parser-8.9.0.jar:/usr/share/logstash/logstash-core/lib/jars/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-1.2-api-2.17.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-api-2.17.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.17.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-jcl-2.17.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-slf4j-impl-2.17.1.jar:/usr/share/logstash/logstash-core/lib/jars/logstash-core.jar:/usr/share/logstash/logstash-core/lib/jars/reflections-0.10.2.jar:/usr/share/logstash/logstash-core/lib/jars/slf4j-api-1.7.32.jar:/usr/share/logstash/logstash-core/lib/jars/snakeyaml-1.33.jar org.logstash.Logstash --path.settings=/usr/share/logstash/config -f /usr/share/logstash/conf.d/ -w 2

The startup.sh is a wrapper which runs the logstash commandline

4

Change log.level: debug or fatal and config.debug: true in logstash.yml. You will see more details what is happening on the startup. The log default location is: /var/log/logstash or logstash/logs if is .tar or .zip.
Also, try to add param -l /usr/share/logstash/logs1 to force your location.

-l, --path.logs PATH
Directory to write Logstash internal logs to.
--log.level LEVEL

Set the log level for Logstash. Possible values are:

  • fatal: log very severe error messages that will usually be followed by the application aborting
  • error: log errors
  • warn: log warnings
  • info: log verbose info (this is the default)
5

From the documentation:

Under Docker, Logstash logs go to standard output by default. To change this behaviour, use any of the techniques above to replace the file at /usr/share/logstash/config/log4j2.properties .

6

@Rios I appreciate your time to answer.

I have enabled debug logs and as well pass the log location in the command line.

I m attaching the logstash debug logs.

Could you please let me know what are the changes required in log4j2.properties.

bash-4.4$ cat /usr/share/logstash/config/log4j2.properties 
status = error
name = LogstashPropertiesConfig

appender.console.type = Console
appender.console.name = plain_console
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c]%notEmpty{[%X{pipeline.id}]}%notEmpty{[%X{plugin.id}]} %m%n

appender.json_console.type = Console
appender.json_console.name = json_console
appender.json_console.layout.type = JSONLayout
appender.json_console.layout.compact = true
appender.json_console.layout.eventEol = true

rootLogger.level = ${sys:ls.log.level}
rootLogger.appenderRef.console.ref = ${sys:ls.log.format}_console
7

Here is full.

Copy from the line 15 to 64
appender.rolling.type = RollingFile
...
rootLogger.appenderRef.routing.ref = pipeline_routing_appender

Also add in logstash.yml
log.format: plain #or json format

8

@Rios Thank you so much for helping me understand.

I m happy to see these files.

bash-4.4$ ls -ltra
total 4032
drwxrwxr-x 1 logstash root          59 Sep 13 16:12 ..
-rw-r--r-- 1 logstash logstash       0 Sep 13 16:12 logstash-json.log
drwxr-xr-x 2 logstash logstash      57 Sep 13 16:12 .
-rw-r--r-- 1 logstash logstash 3105122 Sep 13 16:13 logstash-plain.log

Can you help me with this error.

2023-09-13 16:14:14,573 main ERROR Unable to locate appender "plain_console" for logger config "root"
9

I added this as well to get rid of the error.

Thanks again for helping.

appender.console.type = Console
appender.console.name = plain_console
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c]%notEmpty{[%X{pipeline.id}]}%notEmpty{[%X{plugin.id}]} %m%n
1 Like
10

You don't need both logs, use plain or json format, whatever you prefer.

11

Sure. Will make changes. Thank you for all the help @Rios

1 Like
12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.