Working logstash 7.5.2 upgraded to 7.6.0 causes 3 of my pipelines to fail with syntax error messages.
Or course, since they were working, I can't really find any syntax errors.
Has anyone else seen this?
Thanks
1 Like
What errors are you getting?
Various syntax errors, for example
/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in
execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:326:inblock in converge_state'"]}
[2020-02-25T13:27:47,695][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:filebeat, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \t\r\n], "#", "if", [A-Za-z0-9_-], '"', "'", "}" at line 2, column 1 (byte 10) after filter {\n", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:47:incompile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:55:incompile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:17:inblock in compile_sources'", "org/jruby/RubyArray.java:2580:inmap'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:14:incompile_sources'", "org/logstash/execution/AbstractPipelineExt.java:161:ininitialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:27:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:326:inblock in converge_state'"]}
The config would be this:
#
#
#
input {
beats {
port => 5045
id => "filebeat tcp/5045"
ssl => false
}
}
filter {
if ([fields][app_id] == "something") {
if [message] =~ /^#/ {
drop {
id => "aims index - drop csv header lines, if any"
}
}
}.......
Where the input would be in file 01-inpuy.yml, the filter line is in 10-filter.yml, the if thru the rest is in a 12-something.yml, then other files make up the rest.
This let me write filter sections independently of each other and from the input/output sections. File names are invented to control the concatenation order.
It appears to be objecting to the whitespace that precedes the "if". That's odd.
I have these files in the pipeline config path:
01-input.yml
10-filter.yml
12-aims.yml
16-peoplesoft.yml
19-xx-backstop.yml
30-output.yml
The 1xxx files are all filters, I replaced them with a file created by
cat 1* >> 20-filter.yml
And the pipeline starts like it did in 7.5.2.
Your config files have a .yml extension?
You could try running with --config.debug --log.level debug --config.test_and_exit on the command line and see if the configuration looks the way you expect when using multiple files.
Yea, that embarrassment goes back years because whatever editor I was using highlighted the syntax where .conf did nothing. Extensions are meaningless in Linux anyway 
Is there a trick to config.debug when using pipelines.yml, I get this error
multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
So it won't show me the pipelines as it would try to use them.
Apparently there is a difference between -f and --path.settings... I read that doc syntax as "these are alternatives".
So I get errors with the config check:
[2020-02-25T16:21:16,040][FATAL][logstash.runner ] The given configuration is invalid. Reason: Expected one of [ \t\r\n], "#", "if", [A-Za-z0-9_-], '"', "'", "}" at line 2, column 1 (byte 10) after filter {
It doesn't tell me which pipeline gets the error, but when I try to start logstash, it's in the filebeat pipeline, that section from the config:
[2020-02-25T16:21:15,803][DEBUG][logstash.config.pipelineconfig] -------- Logstash Config ---------
[2020-02-25T16:21:15,805][DEBUG][logstash.config.pipelineconfig] Config from source {:source=>LogStash::Config::Source::MultiLocal, :pipeline_id=>:filebeat}
[2020-02-25T16:21:15,807][DEBUG][logstash.config.pipelineconfig] Config string {:protocol=>"file", :id=>"/etc/logstash/pipelines/filebeat.d/01-input.yml"}
[2020-02-25T16:21:15,808][DEBUG][logstash.config.pipelineconfig]
#
#
#
input {
beats {
port => 5045
id => "filebeat tcp/5045"
ssl => false
}
}
[2020-02-25T16:21:15,810][DEBUG][logstash.config.pipelineconfig] Config string {:protocol=>"file", :id=>"/etc/logstash/pipelines/filebeat.d/10-filter.yml"}
[2020-02-25T16:21:15,811][DEBUG][logstash.config.pipelineconfig]
filter {
[2020-02-25T16:21:15,813][DEBUG][logstash.config.pipelineconfig] Config string {:protocol=>"file", :id=>"/etc/logstash/pipelines/filebeat.d/12-aims.yml"}
[2020-02-25T16:21:15,814][DEBUG][logstash.config.pipelineconfig]
if ([fields][app_id] == "aims") {
if [message] =~ /^#/ {
drop {
id => "aims index - drop csv header lines, if any"
}
}
}
[2020-02-25T16:21:15,816][DEBUG][logstash.config.pipelineconfig] Config string {:protocol=>"file", :id=>"/etc/logstash/pipelines/filebeat.d/16-peoplesoft.yml"}
[2020-02-25T16:21:15,818][DEBUG][logstash.config.pipelineconfig]
#
# PeopleSoft logs
#
if [fields][app_id] == "PeopleSoftApp" {
mutate {
id => "es-qa fix mixed case app_id"
replace => { "[fields][app_id]" => "peoplesoft" }
}
}
So back to the end of the "filter {" line.
-f is equivalent to path.config
Anyways, I think it is objecting to the "filter {" not being closed within the same file. That would be consistent with it working once you concatenate all the files. I wonder if the compiler has been changed to work one file at a time rather than concatenating all the files into a single source.
If you set --java_execution false does the problem go away? I am wondering if this PR to change the compilation phase of java execution has introduced the problem. I do not understand what the PR is doing so that is just a guess since it is the only thing in the release notes that sounds like it could be related.
A workaround would be to open and close filter {} in the configuration for each type of log. It is not a problem to have multiple filter sections.
It's puzzling for sure, I always open the filter in an early file and close it at the end because basically, a lot of these are if.. else if. else if... else logic, each in a separate file. They are pretty big files, I picked the smallest example to debug.
If/else logic can't be outside a filter or span filter files can it?
I'll try the other thing tomorrow, thanks.
--java-execution doesn't seem to matter.
It looks like if a filter {...} is completely contained in a file, it's OK.
I think can do that if it ends up being a breaking change, but it causes a few extra tests and I have to use some other field logic to catch the last "otherwise" where an event doesn't match any existing logic.
But if it's a bug, I won't have too 
In 7.6.0, changes were introduced to add source-file metadata to nodes (elastic/logstash#11456), which was intended to provide greater visibility when debugging pipelines that were composed of many source files. An untended consequence of this is that by parsing the files individually instead of pre-concatenating them, we no longer could parse configurations whose individual files weren't syntactically complete (elastic/logstash#11598). Work is ongoing to find a solution that gets us this source-file metadata in a manner that restores in-the-wild composition styles.
I think I'm going to be able to reformat my files so each is a complete filter {...}.
There are some minor downsides,
- after I matched an "if", the rest of the pipeline was skipped due the else logic
- the last "else" case was entered if none of the above matched, I'll have to set a field on match and test for ![field] to trigger the last else.
We propagate our "self taught" ways to do things developers never imagined.
If 11456 was merged into 7.6.0 shouldn't it have been mentioned in the 7.6.0 release notes? Are the release notes intended to be comprehensive or just the major changes?
1 Like
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.