I'm trying to install logstash as part of an requirement for wazuh

satiswaran · April 22, 2024, 7:54am

I'm trying to install logstash to forward logs to my wazuh server from the wazuh agent for windows, i followed the guide given here: -documentation.wazuh.com/current/cloud-service/your-environment/send-syslog-data

After running the C:\logstash\bin\logstash.bat -f C:\logstash\config\logstash.conf command on the CMD, it returned me with few errors

Command and error:

C:\Users\user>C:\logstash\bin\logstash.bat -f C:\logstash\config\logstash.conf
"Using bundled JDK: C:\logstash\jdk\bin\java.exe"
C:/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_int
C:/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_f
Sending Logstash logs to C:/logstash/logs which is now configured via log4j2.properties
[2024-04-22T15:42:08,003][INFO ][logstash.runner ] Log4j configuration path used is: C:\logstash\config\log4j2.properties
[2024-04-22T15:42:08,003][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.13.2", "jruby.version"=>"jruby 9.4.5.0 (3.1.4) 2023-11-02 1abae2700f OpenJDK 64-Bit Server VM 17.0.10+7 on 17.0.10+7 +indy +jit [x86_64-mswin32]"}
[2024-04-22T15:42:08,003][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dlogstash.jackson.stream-read-constraints.max-string-length=200000000, -Dlogstash.jackson.stream-read-constraints.max-number-length=10000, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11]
[2024-04-22T15:42:08,003][INFO ][logstash.runner ] Jackson default value override logstash.jackson.stream-read-constraints.max-string-length configured to 200000000
[2024-04-22T15:42:08,003][INFO ][logstash.runner ] Jackson default value override logstash.jackson.stream-read-constraints.max-number-length configured to 10000
[2024-04-22T15:42:08,050][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2024-04-22T15:42:09,484][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \t\r\n], "#", [A-Za-z0-9_-], '"', "'", [A-Za-z_], "-", [0-9], "[", "{" at line 3, column 15 (byte 35) after input {\n syslog {\n port => ", :backtrace=>["C:/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:239:in `initialize'", "org/logstash/execution/AbstractPipelineExt.java:173:in `initialize'", "C:/logstash/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "org/jruby/RubyClass.java:931:in `new'", "C:/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:49:in `execute'", "C:/logstash/logstash-core/lib/logstash/agent.rb:386:in `block in converge_state'"]}
[2024-04-22T15:42:09,563][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2024-04-22T15:42:09,578][INFO ][logstash.runner ] Logstash shut down.
[2024-04-22T15:42:09,578][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:808) ~[jruby.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:767) ~[jruby.jar:?]
at C_3a_.logstash.lib.bootstrap.environment.(C:\logstash\lib\bootstrap\environment.rb:90) ~[?:?]

leandrojmp · April 22, 2024, 12:02pm

[2024-04-22T15:42:09,484][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \t\r\n], "#", [A-Za-z0-9_-], '"', "'", [A-Za-z_], "-", [0-9], "[", "{" at line 3, column 15 (byte 35) after input {\n syslog {\n por

You have some configuration error in your file, check it and look at specified line and column.

satiswaran · April 23, 2024, 3:01am

Hi leandrojmp,

[2024-04-23T11:02:12,662][WARN ][logstash.inputs.syslog ][main][c8c0f1ced6acbaa020a12ce8d197ca583f7f8b58edfd57d501de3a006d6b7988] syslog listener died {:protocol=>:udp, :address=>"0.0.0.0:514", :exception=>#<Errno::EADDRINUSE: Address already in use - bind - Address already in use: bindbind(2) for "0.0.0.0" port 514>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:201:in bind'", "C:/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-syslog-3.7.0/lib/logstash/inputs/syslog.rb:191:in udp_listener'", "C:/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-syslog-3.7.0/lib/logstash/inputs/syslog.rb:172:in server'", "C:/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-syslog-3.7.0/lib/logstash/inputs/syslog.rb:152:in block in run'"]}

It shows that its been block in run, i had allow the port on the windows firewall for both inbound and outbound connections, any specific app that needs to be allowed?

satiswaran · April 23, 2024, 5:17am

Hi, i actually managed to resolve it after changing the port number, port 514 was being used by another instance.

C:\Users\user>C:\logstash\bin\logstash.bat "Using bundled JDK: C:\logstash\jdk\bin\java.exe"
C:/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent- C:/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent- Sending Logstash logs to C:/logstash/logs which [2024-04-23T12:55:32,106][INFO ][logstash.runner [2024-04-23T12:55:32,122][INFO ][logstash.runner [2024-04-23T12:55:32,122][INFO ][logstash.runner [2024-04-23T12:55:32,137][INFO ][logstash.runner [2024-04-23T12:55:32,137][INFO ][logstash.runner [2024-04-23T12:55:32,262][WARN ][logstash.config.source.multilocal] [2024-04-23T12:55:40,183][INFO ][logstash.agent [2024-04-23T12:55:40,397][INFO ][org.reflections.Reflections] [2024-04-23T12:55:43,513][INFO ][logstash.javapipeline [2024-04-23T12:55:43,592][INFO ][logstash.javapipeline [2024-04-23T12:55:45,638][INFO ][logstash.javapipeline [2024-04-23T12:55:45,654][WARN ][logstash.filters.grok [2024-04-23T12:55:46,667][INFO ][logstash.javapipeline [2024-04-23T12:55:46,692][INFO ][logstash.inputs.syslog [2024-04-23T12:55:46,692][INFO ][logstash.inputs.syslog [2024-04-23T12:55:46,713][INFO ][logstash.agent [2024-04-23T12:55:49,242][INFO ][logstash.outputs.file [2024-04-23T12:56:41,720][INFO ][logstash.outputs.file [2024-04-23T12:56:48,850][INFO ][logstash.outputs.file [2024-04-23T12:57:36,726][INFO ][logstash.outputs.file [2024-04-23T12:57:59,280][INFO ][logstash.outputs.file [2024-04-23T12:59:11,762][INFO ][logstash.outputs.file [2024-04-23T12:59:12,620][INFO ][logstash.outputs.file [2024-04-23T13:00:21,749][INFO ][logstash.outputs.file [2024-04-23T13:00:23,160][INFO ][logstash.outputs.file [2024-04-23T13:01:21,711][INFO ][logstash.outputs.file [2024-04-23T13:01:33,254][INFO ][logstash.outputs.file [2024-04-23T13:07:26,748][INFO ][logstash.outputs.file [2024-04-23T13:07:43,067][INFO ][logstash.outputs.file [2024-04-23T13:09:21,748][INFO ][logstash.outputs.file [2024-04-23T13:09:25,063][INFO ][logstash.outputs.file [2024-04-23T13:09:51,723][INFO ][logstash.outputs.file [2024-04-23T13:09:55,689][INFO ][logstash.outputs.file [2024-04-23T13:11:11,709][INFO ][logstash.outputs.file [2024-04-23T13:11:15,654][INFO ][logstash.outputs.file [2024-04-23T13:12:11,742][INFO ][logstash.outputs.file [2024-04-23T13:12:27,429][INFO ][logstash.outputs.file [2024-04-23T13:13:21,738][INFO ][logstash.outputs.file [2024-04-23T13:13:38,017][INFO ][logstash.outputs.file [2024-04-23T13:14:31,748][INFO ][logstash.outputs.file [2024-04-23T13:14:39,755][INFO ][logstash.outputs.file [2024-04-23T13:15:41,722][INFO ][logstash.outputs.file [2024-04-23T13:15:59,207][INFO ][logstash.outputs.file [2024-04-23T13:17:01,730][INFO ][logstash.outputs.file [2024-04-23T13:17:10,898][INFO ][logstash.outputs.file [2024-04-23T13:17:46,740][INFO ][logstash.outputs.file [2024-04-23T13:17:50,011][INFO ][logstash.outputs.file [2024-04-23T13:18:11,709][INFO ][logstash.outputs.file [2024-04-23T13:18:29,348][INFO ][logstash.outputs.file [2024-04-23T13:18:51,750][INFO ][logstash.outputs.file [2024-04-23T13:18:59,072][INFO ][logstash.outputs.file -f C:\logstash\config\logstash.conf
ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_int
ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_f
is now configured via log4j2.properties
] Log4j configuration path used is: C:\logstash\config\log4j2.properties
] Starting Logstash {"logstash.version"=>"8.13.2", "jruby.version"=>"jruby 9.4.5.0 (3.1.4) 2023-11-02 1abae2700f OpenJDK 64-Bit Server VM 17.0.10+7 on 17.0.10+7 +indy +jit [x86_64-mswin32]"}
] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dlogstash.jackson.stream-read-constraints.max-string-length=200000000, -Dlogstash.jackson.stream-read-constraints.max-number-length=10000, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11]
] Jackson default value override logstash.jackson.stream-read-constraints.max-string-length configured to 200000000
] Jackson default value override logstash.jackson.stream-read-constraints.max-number-length configured to 10000
Ignoring the 'pipelines.yml' file because modules or command line options are specified
] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
Reflections took 423 ms to scan 1 urls, producing 132 keys and 468 values
] Pipeline main is configured with pipeline.ecs_compatibility: v8 setting. All plugins in this pipeline will default to ecs_compatibility => v8 unless explicitly configured otherwise.
][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["C:/logstash/config/logstash.conf"], :thread=>"#<Thread:0x9dd047f C:/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
][main] Pipeline Java execution initialization time {"seconds"=>2.03}
][main] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
][main] Pipeline started {"pipeline.id"=>"main"}
][main][072b5f42538bff5319d98cf466e3e0b61c84eac10faf6da782290088137682d3] Starting syslog udp listener {:address=>"0.0.0.0:5143"}
][main][072b5f42538bff5319d98cf466e3e0b61c84eac10faf6da782290088137682d3] Starting syslog tcp listener {:address=>"0.0.0.0:5143"}
] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Opening file {:path=>"C:/logstash/logs/KLFirewall.log"}
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Closing file C:/logstash/logs/KLFirewall.log
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Opening file {:path=>"C:/logstash/logs/KLFirewall.log"}
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Closing file C:/logstash/logs/KLFirewall.log
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Opening file {:path=>"C:/logstash/logs/KLFirewall.log"}
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Closing file C:/logstash/logs/KLFirewall.log
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Opening file {:path=>"C:/logstash/logs/KLFirewall.log"}
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Closing file C:/logstash/logs/KLFirewall.log
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Opening file {:path=>"C:/logstash/logs/KLFirewall.log"}
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Closing file C:/logstash/logs/KLFirewall.log
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Opening file {:path=>"C:/logstash/logs/KLFirewall.log"}
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Closing file C:/logstash/logs/KLFirewall.log
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Opening file {:path=>"C:/logstash/logs/KLFirewall.log"}
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Closing file C:/logstash/logs/KLFirewall.log
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Opening file {:path=>"C:/logstash/logs/KLFirewall.log"}
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Closing file C:/logstash/logs/KLFirewall.log
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Opening file {:path=>"C:/logstash/logs/KLFirewall.log"}
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Closing file C:/logstash/logs/KLFirewall.log
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Opening file {:path=>"C:/logstash/logs/KLFirewall.log"}
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Closing file C:/logstash/logs/KLFirewall.log
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Opening file {:path=>"C:/logstash/logs/KLFirewall.log"}
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Closing file C:/logstash/logs/KLFirewall.log
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Opening file {:path=>"C:/logstash/logs/KLFirewall.log"}
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Closing file C:/logstash/logs/KLFirewall.log
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Opening file {:path=>"C:/logstash/logs/KLFirewall.log"}
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Closing file C:/logstash/logs/KLFirewall.log
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Opening file {:path=>"C:/logstash/logs/KLFirewall.log"}
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Closing file C:/logstash/logs/KLFirewall.log
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Opening file {:path=>"C:/logstash/logs/KLFirewall.log"}
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Closing file C:/logstash/logs/KLFirewall.log
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Opening file {:path=>"C:/logstash/logs/KLFirewall.log"}
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Closing file C:/logstash/logs/KLFirewall.log
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Opening file {:path=>"C:/logstash/logs/KLFirewall.log"}
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Closing file C:/logstash/logs/KLFirewall.log
][main][f128fe2905e749db794643ff29a3879e3d32a22b5d4394f7aa2a3bfc117c4a38] Opening file {:path=>"C:/logstash/logs/KLFirewall.log"}