LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"if\", [A-Za-z0-9_-], '\"', \"'\", \"}\"

[Queren_Santos]

I'm having trouble understanding my code configuration error. Shows the following message when trying to run the logs and their settings:

[ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \t\r\n], "#", "if", [A-Za-z0-9_-], '"', "'", "}" at line 3, column 1 (byte 10) after input {\n\n", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:199:in initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:381:in block in converge_state'"]}

[Badger]

What does your configuration look like? The compiler is objecting to whatever comes after

input {

So tell us what comes after that.

[cheshirecat (Krzysztof Smaga)]

Please paste (with </> tag) your logstash.yml and files from conf.d/*.yml.

[Queren_Santos]

input {
file {
path => "/tmp/spreadsheet_data.csv"
start_position => "beginning"
sincedb_path => "NULL"
}
}
filter {
csv {
separator => ","
columns => ["Name","Class","Dorm","Room","GPA"]
}
}
output {
elasticsearch {
hosts =>"http://localhost:9200"
index =>"spreadsheet_data"
}
}

[Queren_Santos]

What would the syntax look like?

[Badger]

@Queren_Santos

sincedb_path => "NULL"

That will create a file called NULL in logstash's working directory. If you do not want the in-memory sincedb persisted across restarts then use sincedb_path => "/dev/null".

[Queren_Santos]

Thanks for the suggestion, but unfortunately it didn't work. The filter is basic with csv to json conversion, I want to analyze this data in Grafana. It shows an error message in the output part of the code, do you have any idea how the syntax of this part would be?

Error message:

[FATAL] 2022-08-23 11:36:28.419 [LogStash::Runner] runner - The given configuration is invalid. Reason: Expected one of [A-Za-z0-9_-], [ \t\r\n], "#", "{" at line 19, column 17 (byte 356) after output {
output

[Badger]

Again, that would entirely depend on what comes after the text in the error message.

[Queren_Santos]

This is the error

• Ignoring the 'pipelines.yml' file because modules or command line options are specified

[Queren_Santos]

File pipeline.yml

This file is where you define your pipelines. You can define multiple.

For more information on multiple pipelines, see the documentation:

Multiple Pipelines | Logstash Reference [8.3] | Elastic

• pipeline.id: main

path.config: "/etc/logstash/conf.d/*.conf"

path.config: "/etc/logstash/*.config"

[cheshirecat (Krzysztof Smaga)]

@Queren_Santos

input {
file {
path => "/tmp/spreadsheet_data.csv"
start_position => "beginning"
sincedb_path => "NULL"
}
}
filter {
csv {
separator => ","
columns => ["Name","Class","Dorm","Room","GPA"]
}
}
output {
elasticsearch {
hosts =>"http://localhost:9200"
index =>"spreadsheet_data"
}
}

First thing - please remember that YAML uses spaces. Please change this:

@Queren_Santos

hosts =>"http://localhost:9200"
index =>"spreadsheet_data"

into this:

hosts => "http://localhost:9200"
index => "spreadsheet_data"

[cheshirecat (Krzysztof Smaga)]

@Queren_Santos

Ignoring the 'pipelines.yml' file because modules or command line options are specified

This message says that you should disable whole this section:

# ------------ Pipeline Settings --------------
#
# The ID of the pipeline.
#
#pipeline.id: main
#
# Set the number of workers that will, in parallel, execute the filters+outputs
# stage of the pipeline.
#
# This defaults to the number of the host's CPU cores.
#
#pipeline.workers: 2
#
# How many events to retrieve from inputs before sending to filters+workers
#
#pipeline.batch.size: 125
#
# How long to wait in milliseconds while polling for the next event
# before dispatching an undersized batch to filters+outputs
#
#pipeline.batch.delay: 50
#
# Force Logstash to exit during shutdown even if there are still inflight
# events in memory. By default, logstash will refuse to quit until all
# received events have been pushed to the outputs.
#
# WARNING: enabling this can lead to data loss during shutdown
#
# pipeline.unsafe_shutdown: false
#
# Set the pipeline event ordering. Options are "auto" (the default), "true" or "false".
# "auto" will  automatically enable ordering if the 'pipeline.workers' setting
# is also set to '1'.
# "true" will enforce ordering on the pipeline and prevent logstash from starting
# if there are multiple workers.
# "false" will disable any extra processing necessary for preserving ordering.
#
#pipeline.ordered: auto
#
# Sets the pipeline's default value for `ecs_compatibility`, a setting that is
# available to plugins that implement an ECS Compatibility mode for use with
# the Elastic Common Schema.
# Possible values are:
# - disabled (default)
# - v1
# - v8
# The default value will be `v8` in Logstash 8, making ECS on-by-default. To ensure a
# migrated pipeline continues to operate as it did before your upgrade, opt-OUT
# of ECS for the individual pipeline in its `pipelines.yml` definition. Setting
# it here will set the default for _all_ pipelines, including new ones.
#
# pipeline.ecs_compatibility: disabled

Than in section Pipeline Configuration Settings you should enable this setting:

path.config: "/etc/logstash/conf.d/*.conf"

and after that you can adjust other settings in this section. Please remember that your conf file should be located in /etc/logstash/conf.d/ directory. You can use more than one conf file and have more that one input/output setting.

[Queren_Santos]

Thanks, that really was it, setup ok