I'm having trouble testing a grok filter. I've been able to run a very basic test using the file input and then stdout output. All I'm trying to do is run a test with a grok filter that takes a file (oneline.txt), utilizes the grok filter and then sends the file to stdout. I'm stuck at the stdout part. When I run the configuration nothing displays on the screen as the config fails at output { stdout. After speaking with a colleague he mentioned its probably that the grok pattern is failing.
So two questions:
How do I allow my first and basic test to keep printing to stdout after I've run the config file once? After reading the docs and forum posts I thought the sincedb_path null line took care of that.
If stdout is working, as proved with the first test, why is the grok config failing? I'm going to side on the user error side of things on this one.
helloworld.txt located in /tmp
Hello World!
helloworld-pipeline.conf
input {
file {
path => "/tmp/helloworld.txt"
start_position => "beginning"
sincedb_path => "nul"
}
}
# The filter part of this file is commented out to indicate that it is
# optional.
#filter {
#
#}
output {
stdout {}
}
cmd that works and prints to stdout. : sudo bin/logstash -f /usr/share/logstash/helloworld-pipeline.conf --path.settings /etc/logstash
{
"@version" => "1",
"@timestamp" => 2020-03-06T14:41:43.392Z,
"message" => "Hello World!!!",
"host" => "dch1095ql5app.svc.ny.gov",
"path" => "/tmp/helloworld.txt"
}
oneline.log located in /tmp
T: 2019-09-30 14:11:14,057 |L: INFO |MSG: Start- upsert of allocated status from service for holding date:10/19/2019 4:00:00 AM
first-pipeline.conf
# The # character at the beginning of a line indicates a comment. Use
# comments to describe your configuration.
input {
file {
path => "/tmp/oneline.log"
start_position => "beginning"
sincedb_path => "nul"
}
}
# The filter part of this file is commented out to indicate that it is
# optional.
filter {
grok {
match => { "message" => "\T\W\s*%{TIMESTAMP_ISO8601:timestamp}\s\|L\:\s*%{LOGLEVEL:log-level}\s\|MSG\:\s*%{GREEDYDATA:message}" }
}
output {
stdout {}
}
Run cmd from /usr/share/logstash:
sudo bin/logstash -f /usr/share/logstash/first-pipeline.conf --path.settings /etc/logstash
Response:
sudo bin/logstash -f /usr/share/logstash/first-pipeline.conf --path.settings /etc/logstash
Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2020-03-06T10:43:49,711][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-03-06T10:43:50,071][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.5.1"}
[2020-03-06T10:43:52,443][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"=>\" at line 18, column 10 (byte 484) after filter {\n grok {\n match => { \"message\" => \"\\T\\W\\s*%{TIMESTAMP_ISO8601:timestamp}\\s\\|L\\:\\s*%{LOGLEVEL:log-level}\\s\\|MSG\\:\\s*%{GREEDYDATA:message}\" }\n}\n\noutput {\n stdout ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2584:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:156:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:27:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:326:in `block in converge_state'"]}