Error in logstash plain file

michaeljsamuel · December 6, 2022, 9:49pm

Hi there,
I am trying to use logstash on windows and believe I have downloaded it correctly however i am unable to forward any logs and this is what is in my plain file

[2022-12-07T10:41:03,853][INFO ][logstash.runner ] Log4j configuration path used is: C:\logstash-8.5.2\config\log4j2.properties [2022-12-07T10:41:03,857][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.5.2", "jruby.version"=>"jruby 9.3.8.0 (2.6.8) 2022-09-13 98d69c9461 OpenJDK 64-Bit Server VM 17.0.5+8 on 17.0.5+8 +indy +jit [x86_64-mswin32]"} [2022-12-07T10:41:03,857][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED] [2022-12-07T10:41:03,930][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2022-12-07T10:41:05,677][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false} [2022-12-07T10:41:05,802][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"input\", \"filter\", \"output\" at line 13, column 1 (byte 188) after ", :backtrace=>["C:/logstash-8.5.2/logstash-core/lib/logstash/compiler.rb:32:in compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:182:in initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in initialize'", "C:/logstash-8.5.2/logstash-core/lib/logstash/java_pipeline.rb:48:in initialize'", "org/jruby/RubyClass.java:911:in new'", "C:/logstash-8.5.2/logstash-core/lib/logstash/pipeline_action/create.rb:50:in execute'", "C:/logstash-8.5.2/logstash-core/lib/logstash/agent.rb:386:in block in converge_state'"]}
[2022-12-07T10:41:05,856][INFO ][logstash.runner ] Logstash shut down.
[2022-12-07T10:41:05,856][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:790) ~[jruby.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:753) ~[jruby.jar:?]
at C_3a_.logstash_minus_8_dot_5_dot_2.lib.bootstrap.environment.(C:\logstash-8.5.2\lib\bootstrap\environment.rb:91) ~[?:?]`

Would appreciate any assistance,
TIA

Badger · December 6, 2022, 11:17pm

What does the file that contains the input/filter/output definitions look like?

michaeljsamuel · December 6, 2022, 11:53pm

Thanks for the response, I actually redownloaded it and now find the plain file to be no errors, however still do not receive any data to the interface I am trying to (newrelic)

The .conf file looks like this

input {
  file {
  path => "C:\Logs\logstashtest.txt"
  }
}
# Replace is a valid license key
output {
  newrelic {
  base_uri => "https://log-api.newrelic.com/log/v1"
  license_key => "##license key"
  }
}

Where the logstashtest.log file just has some words in it that I can search for.
The plain file outputs this

[2022-12-07T12:42:55,475][INFO ][logstash.runner          ] Log4j configuration path used is: C:\logstash-8.5.2\config\log4j2.properties
[2022-12-07T12:42:55,481][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.5.2", "jruby.version"=>"jruby 9.3.8.0 (2.6.8) 2022-09-13 98d69c9461 OpenJDK 64-Bit Server VM 17.0.5+8 on 17.0.5+8 +indy +jit [x86_64-mswin32]"}
[2022-12-07T12:42:55,483][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2022-12-07T12:42:55,569][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-12-07T12:42:57,449][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-12-07T12:42:57,777][INFO ][org.reflections.Reflections] Reflections took 95 ms to scan 1 urls, producing 125 keys and 438 values
[2022-12-07T12:42:58,568][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2022-12-07T12:42:58,650][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>12, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1500, "pipeline.sources"=>["C:/logstash-8.5.2/config/logstash.conf"], :thread=>"#<Thread:0x3f3ac28d run>"}
[2022-12-07T12:42:59,407][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.75}
[2022-12-07T12:42:59,455][INFO ][logstash.inputs.file     ][main] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"C:/logstash-8.5.2/data/plugins/inputs/file/.sincedb_8068e9a8a536a697bc390e1ebfb70863", :path=>["C:\\Logs\\logtest.txt"]}
[2022-12-07T12:42:59,470][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-12-07T12:42:59,509][INFO ][filewatch.observingtail  ][main][4dd05b45f1d7c66a603609e0099e2360d24adad19d83ee5cf07ea73035639040] START, creating Discoverer, Watch with file and sincedb collections
[2022-12-07T12:42:59,526][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}

Cheers

Badger · December 7, 2022, 12:43am

Try changing the \ to /. Last time I checked \ was treated as an escape, so it is waiting for something to be written to C:Logslogsstashtest.txt.

If that does not help set --log.level trace and looks for messages from the filewatch module. This thread shows what filewatch traces look like.

michaeljsamuel · December 7, 2022, 8:33pm

Hi, thanks for the reccomendation and I fixed the previous error then changed to debug and now I have this,

[2022-12-08T09:30:37,858][DEBUG][logstash.runner          ] node.uuid: ""
[2022-12-08T09:30:37,858][DEBUG][logstash.runner          ] xpack.management.enabled: false
[2022-12-08T09:30:37,859][DEBUG][logstash.runner          ] xpack.management.logstash.poll_interval: #<Java::OrgLogstashUtil::TimeValue:0x3c9be5f1>
[2022-12-08T09:30:37,859][DEBUG][logstash.runner          ] xpack.management.pipeline.id: ["main"]
[2022-12-08T09:30:37,860][DEBUG][logstash.runner          ] xpack.management.elasticsearch.username: "logstash_system"
[2022-12-08T09:30:37,861][DEBUG][logstash.runner          ] xpack.management.elasticsearch.hosts: ["https://localhost:9200"]
[2022-12-08T09:30:37,863][DEBUG][logstash.runner          ] xpack.management.elasticsearch.ssl.verification_mode: "certificate"
[2022-12-08T09:30:37,863][DEBUG][logstash.runner          ] xpack.management.elasticsearch.sniffing: false
[2022-12-08T09:30:37,865][DEBUG][logstash.runner          ] --------------- Logstash Settings -------------------
[2022-12-08T09:30:37,897][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"C:/logstash-8.5.2/config/pipelines.yml"}
ERROR: Failed to read pipelines yaml file. Location: C:/logstash-8.5.2/config/pipelines.yml
usage:
  bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG]
  bin/logstash --modules MODULE_NAME [-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"] [-t] [-w COUNT] [-l LOG]
  bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
  bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
  bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
  bin/logstash --help
[2022-12-08T09:30:37,954][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:790) ~[jruby.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:753) ~[jruby.jar:?]
        at C_3a_.logstash_minus_8_dot_5_dot_2.lib.bootstrap.environment.<main>(C:\logstash-8.5.2\lib\bootstrap\environment.rb:91) ~[?:?]

Could you recommend any next steps, currently my pipelines.yml is fully commented out

Thanks very much

Badger · December 7, 2022, 9:44pm

That would mean you effectively have no configuration, I would not be surprised if that produced an error. You should configure at least one pipeline.

michaeljsamuel · December 7, 2022, 9:54pm

Yup, I just looked into that and did one pipeline

 - pipeline.id: test
   path.config: "/config/logstash.conf"

and now the errors instead

[2022-12-08T10:52:59,644][FATAL][logstash.runner          ] Logstash could not be started because there is already another instance using the configured data directory.  If you wish to run multiple instances, you must change the "path.data" setting.

And ive added to the logstash.yml file

path.data: "C:/logstash-8.5.2/data"

However no change in the error from debug.
Sorry for all the questions

Badger · December 7, 2022, 10:31pm

It is failing to obtain a lock on .lock in the path.data directory. If there really is not a second copy of logstash running then it could conceivably be a permissions issue.

michaeljsamuel · December 7, 2022, 11:09pm

To my knowledge there is only one logstash running, im trying to just output now into a file so I can ensure the problem is on logstashs end not newrelic. But still I get that error regardless of where I point the path.data

michaeljsamuel · December 8, 2022, 2:01am

Hey again sorry,
Ive been working at debugging it and one of the commands I did to find the version which was

.\logstash --version

responded in

logstash Version not detected

could this be my problem and if so what does this mean?

Thanks very much

Badger · December 8, 2022, 2:11am

Does the problem go away if you reboot the Windows machine? If you set path.data and run logstash what does the directory listing of path.data contain?

michaeljsamuel · December 8, 2022, 2:24am

I restarted the machine
The logs show this

2022-12-08T15:19:08,500][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.5.2", "jruby.version"=>"jruby 9.3.8.0 (2.6.8) 2022-09-13 98d69c9461 OpenJDK 64-Bit Server VM 17.0.5+8 on 17.0.5+8 +indy +jit [x86_64-mswin32]"}
[2022-12-08T15:19:08,500][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2022-12-08T15:19:08,577][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-12-08T15:19:10,477][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-12-08T15:19:10,769][INFO ][org.reflections.Reflections] Reflections took 85 ms to scan 1 urls, producing 125 keys and 438 values
[2022-12-08T15:19:11,555][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2022-12-08T15:19:11,630][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>12, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1500, "pipeline.sources"=>["C:/logstash-8.5.2/config/logstash.conf"], :thread=>"#<Thread:0x7755e4d0 run>"}
[2022-12-08T15:19:12,312][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.68}
[2022-12-08T15:19:12,354][INFO ][logstash.inputs.file     ][main] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"C:/logstash-8.5.2/data/plugins/inputs/file/.sincedb_a634988dbff5570ae797bb95c52a5491", :path=>["C:/Logs/logstashtest.log"]}
[2022-12-08T15:19:12,367][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-12-08T15:19:12,395][INFO ][filewatch.observingtail  ][main][ce4d8c0da3fc9feed869da4272d9177e48199eb7bee47fbf08517d9dad08e886] START, creating Discoverer, Watch with file and sincedb collections
[2022-12-08T15:19:12,405][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}

And the directory contains a .lock file and

michaeljsamuel · December 8, 2022, 2:43am

fyi when i run the same --version command i still get the same output

Badger · December 8, 2022, 2:45am

So it is working now, right?

michaeljsamuel · December 8, 2022, 2:56am

No, in debug mode i still receive this error,

[2022-12-08T15:53:27,398][FATAL][logstash.runner          ] Logstash could not be started because there is already another instance using the configured data directory.  If you wish to run multiple instances, you must change the "path.data" setting.
[2022-12-08T15:53:27,401][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit

and my two instances to my understanding are the pipeline and logstash.yml files however whenever I remove parts of either one of them i get the other error of the failed to read pipelines yaml file

Badger · December 8, 2022, 3:15am

No, that is referring to there being two different processes (possibly from two different users) both trying to run logstash with the same value of path.data.

It worked after the reboot, right? Is it consistent that the first run after a reboot works and subsequent runs do not?

michaeljsamuel · December 8, 2022, 3:22am

I mean by worked, when I put the logs to verbose and info instead of debug I dont get errors but no data goes to newrelic and if I put it into debug I get errors. I ran the config file through the internal config.test_and_exit and it worked correctly. Just unsure why I am not receiving output to newrelic and also as to what the 2 user error is.

Sorry for asking so much

Badger · December 8, 2022, 3:49am

You can see the logstash.bat code here. That suggests that you might have a corrupt install, or an environment variable pointing something to the wrong place. Neither %LS_HOME%\logstash-core\versions-gem-copy.yml (for zip/deb/rpm installs), nor %LS_HOME%\versions.yml (for git checkouts) contains a version string labelled logstash. (LS_HOME gets set by bin/setup.bat.)

For example, my .zip install has a versions-gem-copy.yml that starts with

---
# alpha and beta qualifiers are now added via VERSION_QUALIFIER environment var
logstash: 8.4.3
logstash-core: 8.4.3
logstash-core-plugin-api: 2.1.16

Not sure what else to say.

michaeljsamuel · December 8, 2022, 8:39pm

I have a versions-gem-copy.yml file in you directory you mentioned of /logstash-core where the first sentence is

# alpha and beta qualifiers are now added via VERSION_QUALIFIER environment var
logstash: 8.5.2
logstash-core: 8.5.2
logstash-core-plugin-api: 2.1.16

How could I check that the environment variable is pointing in the wrong place?
Thanks

system · January 5, 2023, 8:39pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.