Problem:
When setting up a Docker-based Elastic Stack (Elasticsearch, Logstash, and Kibana) environment. The Logstash service was not able to start correctly and reported the following error message:
[2023-06-23T16:41:54,749][ERROR][logstash.agent ] Failed to execute action
{:action=>LogStash::PipelineAction::Create/pipeline_id:main,
:exception=>"LogStash::ConfigurationError",
:message=>"Expected one of [ \t\r\n], "#", "input", "filter", "output" at line 1, column 1 (byte 1)",
:backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:187:in initialize'",
"org/logstash/execution/JavaBasePipelineExt.java:72:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in initialize'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:391:in block in converge_state'"]}
The relevant lines from the docker-compose.yml file were:
volumes:
- ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml
- ./logstash/pipeline:/usr/share/logstash/pipeline
And the logstash.yml file specified the path.config as /usr/share/logstash/pipeline.
In the /usr/share/logstash/pipeline directory, we had two files: logstash.conf (the correct pipeline configuration file) and columns.txt (a text file used by the logstash.conf pipeline). This caused Logstash to treat columns.txt as a configuration file because it attempts to parse all files in the path.config directory as pipeline configuration files. Because columns.txt came alphabetically before logstash.conf, Logstash attempted to parse it first and failed, resulting in the error message above.
Solution:
Upon reading the Logstash documentation closely, we found this crucial line:
Every file in the host directory ~/pipeline/ will then be parsed by Logstash as pipeline configuration.
This implied that Logstash was looking for pipeline configuration files directly under the /usr/share/logstash/pipeline directory, and it would not distinguish between different file types.
By moving the columns.txt file out of the /usr/share/logstash/pipeline directory and into the /usr/share/logstash/data directory, we were able to resolve the issue. Logstash started up correctly and was able to process log files as expected, using the logstash.conf pipeline configuration and the columns.txt file as intended.
This experience highlights the importance of understanding how configuration files are loaded by different services in the Elastic Stack, especially when those services are containerized using Docker.
The root cause of the problem was a misunderstanding of the file-loading behavior of Logstash. It treats all files in the path.config directory as configuration files, regardless of their file types or intended uses. This experience could be helpful to others setting up similar environments.