Logstash 8.2.2 on windows 10

Hello,

logstash 8.2.2Preformatted text for windows failed to run, i have the following output.
Could someone please help

Regards

C:\WINDOWS\system32>logstash -f logstash.conf
"Using bundled JDK: D:\Technical-Docs\Audits\Audit-Security\Tools-2022\Elastic\logstash-8.2.2\jdk\bin\java.exe"
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to D:/Technical-Docs/Audits/Audit-Security/Tools-2022/Elastic/logstash-8.2.2/logs which is now configured via log4j2.properties
[2022-06-14T09:23:15,219][INFO ][logstash.runner          ] Log4j configuration path used is: D:\Technical-Docs\Audits\Audit-Security\Tools-2022\Elastic\logstash-8.2.2\config\log4j2.properties
[2022-06-14T09:23:15,227][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.2.2", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.14.1+1 on 11.0.14.1+1 +indy +jit [mswin32-x86_64]"}
[2022-06-14T09:23:15,228][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2022-06-14T09:23:15,308][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-06-14T09:23:16,967][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-06-14T09:23:17,233][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [A-Za-z0-9_-], [ \\t\\r\\n], \"#\", \"{\", [A-Za-z0-9_], \",\", \"]\" at line 7, column 23 (byte 76) after output {\r\n elasticsearch {\r\n   hosts => [localhost", :backtrace=>["D:/Technical-Docs/Audits/Audit-Security/Tools-2022/Elastic/logstash-8.2.2/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:189:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "D:/Technical-Docs/Audits/Audit-Security/Tools-2022/Elastic/logstash-8.2.2/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "D:/Technical-Docs/Audits/Audit-Security/Tools-2022/Elastic/logstash-8.2.2/logstash-core/lib/logstash/pipeline_action/create.rb:50:in `execute'", "D:/Technical-Docs/Audits/Audit-Security/Tools-2022/Elastic/logstash-8.2.2/logstash-core/lib/logstash/agent.rb:381:in `block in converge_state'"]}
[2022-06-14T09:23:17,305][INFO ][logstash.runner          ] Logstash shut down.
[2022-06-14T09:23:17,314][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby.jar:?]
        at D_3a_.Technical_minus_Docs.Audits.Audit_minus_Security.Tools_minus_2022.Elastic.logstash_minus_8_dot_2_dot_2.lib.bootstrap.environment.<main>(D:\Technical-Docs\Audits\Audit-Security\Tools-2022\Elastic\logstash-8.2.2\lib\bootstrap\environment.rb:91) ~[?:?]
C:\WINDOWS\system32>logstash -f logstash.conf
"Using bundled JDK: D:\Technical-Docs\Audits\Audit-Security\Tools-2022\Elastic\logstash-8.2.2\jdk\bin\java.exe"
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to D:/Technical-Docs/Audits/Audit-Security/Tools-2022/Elastic/logstash-8.2.2/logs which is now configured via log4j2.properties
[2022-06-14T09:23:15,219][INFO ][logstash.runner          ] Log4j configuration path used is: D:\Technical-Docs\Audits\Audit-Security\Tools-2022\Elastic\logstash-8.2.2\config\log4j2.properties
[2022-06-14T09:23:15,227][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.2.2", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.14.1+1 on 11.0.14.1+1 +indy +jit [mswin32-x86_64]"}
[2022-06-14T09:23:15,228][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2022-06-14T09:23:15,308][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-06-14T09:23:16,967][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-06-14T09:23:17,233][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [A-Za-z0-9_-], [ \\t\\r\\n], \"#\", \"{\", [A-Za-z0-9_], \",\", \"]\" at line 7, column 23 (byte 76) after output {\r\n elasticsearch {\r\n   hosts => [localhost", :backtrace=>["D:/Technical-Docs/Audits/Audit-Security/Tools-2022/Elastic/logstash-8.2.2/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:189:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "D:/Technical-Docs/Audits/Audit-Security/Tools-2022/Elastic/logstash-8.2.2/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "D:/Technical-Docs/Audits/Audit-Security/Tools-2022/Elastic/logstash-8.2.2/logstash-core/lib/logstash/pipeline_action/create.rb:50:in `execute'", "D:/Technical-Docs/Audits/Audit-Security/Tools-2022/Elastic/logstash-8.2.2/logstash-core/lib/logstash/agent.rb:381:in `block in converge_state'"]}
[2022-06-14T09:23:17,305][INFO ][logstash.runner          ] Logstash shut down.
[2022-06-14T09:23:17,314][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby.jar:?]
        at D_3a_.Technical_minus_Docs.Audits.Audit_minus_Security.Tools_minus_2022.Elastic.logstash_minus_8_dot_2_dot_2.lib.bootstrap.environment.<main>(D:\Technical-Docs\Audits\Audit-Security\Tools-2022\Elastic\logstash-8.2.2\lib\bootstrap\environment.rb:91) ~[?:?]

Hello @Airborn

Welcome to elastic community :slight_smile: !!!

The below error would exactly brief us out that there is issue in your configuration.

[2022-06-14T09:23:17,233][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [A-Za-z0-9_-], [ \\t\\r\\n], \"#\", \"{\", [A-Za-z0-9_], \",\", \"]\" at line 7, column 23 (byte 76) after output {\r\n elasticsearch {\r\n hosts => [localhost", :backtrace=>["D:/Technical-Docs/Audits/Audit-Security/Tools-2022/Elastic/logstash-8.2.2/logstash-core/lib/logstash/compiler.rb:32:in compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:189:in initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in initialize'", "D:/Technical-Docs/Audits/Audit-Security/Tools-2022/Elastic/logstash-8.2.2/logstash-core/lib/logstash/java_pipeline.rb:48:in initialize'", "D:/Technical-Docs/Audits/Audit-Security/Tools-2022/Elastic/logstash-8.2.2/logstash-core/lib/logstash/pipeline_action/create.rb:50:in execute'", "D:/Technical-Docs/Audits/Audit-Security/Tools-2022/Elastic/logstash-8.2.2/logstash-core/lib/logstash/agent.rb:381:in block in converge_state'"]}

Hence, Please check out the below line in the config code

hosts => [localhost",

Example code snippet:

output
{
elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "<Your-index-name>"
    user => "<your-es-username>" #if its enabld with authentication
    password => "<your-es-pwd>" #if its enabld with authentication
}
}

Keep posted with updates!!! Thanks !!! :slight_smile:

Hi

Thanks for replying

bellow is my logstash.conf file, i don't use authentication. I'm using all downloaded files without any change and i'm using Windows 10 version 21H2

I don't have any problems with Elasticsearch and Kibana

Regards

output
{
Elasticsearch {
hosts => ["http://localhost:9200"]
index => "indexforlogstash"
}
}

Hello @Airborn

I believe the "pipeline.id: main" code block in pipelines.yml should be commented out as its not been used.
Could you check the pipeline id called "main" in pipelines.yml

bellow is the content of pipelines.yml

As i'm new, the file is the default one

Regards


List of pipelines to be loaded by Logstash

This document must be a list of dictionaries/hashes, where the keys/values are pipeline settings.

Default values for omitted settings are read from the logstash.yml file.

When declaring multiple pipelines, each MUST have its own pipeline.id.

Example of two pipelines:

- pipeline.id: test

pipeline.workers: 1

pipeline.batch.size: 1

config.string: "input { generator {} } filter { sleep { time => 1 } } output { stdout { codec => dots } }"

- pipeline.id: another_test

queue.type: persisted

path.config: "/tmp/logstash/*.config"

Available options:

# name of the pipeline

pipeline.id: mylogs

# The configuration string to be used by this pipeline

config.string: "input { generator {} } filter { sleep { time => 1 } } output { stdout { codec => dots } }"

# The path from where to read the configuration text

path.config: "/etc/conf.d/logstash/myconfig.cfg"

# How many worker threads execute the Filters+Outputs stage of the pipeline

pipeline.workers: 1 (actually defaults to number of CPUs)

# How many events to retrieve from inputs before sending to filters+workers

pipeline.batch.size: 125

# How long to wait in milliseconds while polling for the next event

# before dispatching an undersized batch to filters+outputs

pipeline.batch.delay: 50

Set the pipeline event ordering. Options are "auto" (the default), "true" # # or "false".

"auto" automatically enables ordering if the 'pipeline.workers' setting

is also set to '1', and disables otherwise.

"true" enforces ordering on a pipeline and prevents logstash from starting

a pipeline with multiple workers allocated.

"false" disable any extra processing necessary for preserving ordering.

pipeline.ordered: auto

# Internal queuing model, "memory" for legacy in-memory based queuing and

# "persisted" for disk-based acked queueing. Defaults is memory

queue.type: memory

# If using queue.type: persisted, the page data files size. The queue data consists of

# append-only data files separated into pages. Default is 64mb

queue.page_capacity: 64mb

# If using queue.type: persisted, the maximum number of unread events in the queue.

# Default is 0 (unlimited)

queue.max_events: 0

# If using queue.type: persisted, the total capacity of the queue in number of bytes.

# Default is 1024mb or 1gb

queue.max_bytes: 1024mb

# If using queue.type: persisted, the maximum number of acked events before forcing a checkpoint

# Default is 1024, 0 for unlimited

queue.checkpoint.acks: 1024

# If using queue.type: persisted, the maximum number of written events before forcing a checkpoint

# Default is 1024, 0 for unlimited

queue.checkpoint.writes: 1024

# If using queue.type: persisted, the interval in milliseconds when a checkpoint is forced on the head page

# Default is 1000, 0 for no periodic checkpoint.

queue.checkpoint.interval: 1000

# Enable Dead Letter Queueing for this pipeline.

dead_letter_queue.enable: false

If using dead_letter_queue.enable: true, the maximum size of dead letter queue for this pipeline. Entries

will be dropped if they would increase the size of the dead letter queue beyond this setting.

Default is 1024mb

dead_letter_queue.max_bytes: 1024mb

If using dead_letter_queue.enable: true, the interval in milliseconds where if no further events eligible for the DLQ

have been created, a dead letter queue file will be written. A low value here will mean that more, smaller, queue files

may be written, while a larger value will introduce more latency between items being "written" to the dead letter queue, and

being available to be read by the dead_letter_queue input when items are are written infrequently.

Default is 5000.

dead_letter_queue.flush_interval: 5000

If using dead_letter_queue.enable: true, the directory path where the data files will be stored.

Default is path.data/dead_letter_queue

path.dead_letter_queue:

Hello @Airborn

As you are not using any pipelines , i would request you to comment all the lines in "pipelines.yml" save it. Then try executing the command.

Keep Posted!!! Thanks !!!

ps: Also please change the tags on this question

just to add that all the lines in pipelines.yml file i'm using are commented (prefixed by Preformatted text # )

As i see,it seems that Preformatted text# has disappeared when i pasted in the post

@Airborn
Could you share the logstash.conf code

Bellow is the logstash.conf file

output
{
Elasticsearch {
hosts => ["http://localhost:9200"]
index => "indexforlogstash"
}
}

Hello @Airborn ,

There is a typo in your logstash.conf file

Error: "E" in Elasticsearch
Corrected code syntax:

output
{
elasticsearch {
hosts => ["http://localhost:9200"]
index => "indexforlogstash"
}
}

if you would like to parse your input to index, you can try the below

input { 
stdin { 
} 

}

output
{
elasticsearch {
hosts => ["http://localhost:9200"]
index => "indexforlogstash"
}
}

Keep Posted with updates !!! Thanks and Happy Stashing :slight_smile: !!!

https://www.elastic.co/guide/en/logstash/current/configuration.html

I would also request you to mention the input plugin as it is mandatory like the output plugin.
filter plugin is optional.

I did the change and i have the same problem.

for info, i got the .conf file content from Configuring Logstash | Logstash Reference [8.2] | Elastic

what does the following error message means

[2022-06-14T12:53:07,766][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [A-Za-z0-9_-], [ \t\r\n], "#", "{", [A-Za-z0-9_], ",", "]" at line 7, column 23 (byte 76) after output {\r\n Elasticsearch {\r\n hosts => [localhost", :backtrace=>["D:/Technical-Docs/Audits/Audit-Security/Tools-2022/Elastic/logstash-8.2.2/logstash-core/lib/logstash/compiler.rb:28:in compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:189:in initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in initialize'", "D:/Technical-Docs/Audits/Audit-Security/Tools-2022/Elastic/logstash-8.2.2/logstash-core/lib/logstash/java_pipeline.rb:48:in initialize'", "D:/Technical-Docs/Audits/Audit-Security/Tools-2022/Elastic/logstash-8.2.2/logstash-core/lib/logstash/pipeline_action/create.rb:50:in execute'", "D:/Technical-Docs/Audits/Audit-Security/Tools-2022/Elastic/logstash-8.2.2/logstash-core/lib/logstash/agent.rb:381:in block in converge_state'"]}

@Airborn

i could still see that typo error in this error message "E" in "Elasticsearch"

Please save the changes and try executing once.

You are right. now it is working
thank you so much for you valuable help and support

1 Like

Thanks a lot for your feedback.

Happy Stashing!!! :slight_smile: