Using the most basic configuration below (I've removed the more complex match patterns):
input {
lumberjack {
port => 6782
ssl_certificate => '/etc/pki/tls/certs/logstash-forwarder.crt'
ssl_key => '/etc/pki/tls/private/logstash-forwarder.key'
}
}
filter {
grok {
match => [ 'file', "%{PATH}/%{NOTSPACE}.l%{NOTSPACE}" ]
}
}
output {
stdout { codec => rubydebug }
}
Generates the following output:
==> logstash.log <==
{:timestamp=>"2015-11-04T12:12:18.324000+0000", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Lumberjack input", :exception=>LogStash::SizedQueueTimeout::TimeoutError, :level=>:warn}
{:timestamp=>"2015-11-04T12:12:18.335000+0000", :message=>"Lumberjack input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::CircuitBreaker::HalfOpenBreaker, :level=>:warn}
{:timestamp=>"2015-11-04T12:12:23.544000+0000", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Lumberjack input", :exception=>LogStash::SizedQueueTimeout::TimeoutError, :level=>:warn}
{:timestamp=>"2015-11-04T12:12:23.550000+0000", :message=>"Lumberjack input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::CircuitBreaker::HalfOpenBreaker, :level=>:warn}
{:timestamp=>"2015-11-04T12:12:28.706000+0000", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Lumberjack input", :exception=>LogStash::SizedQueueTimeout::TimeoutError, :level=>:warn}
{:timestamp=>"2015-11-04T12:12:28.714000+0000", :message=>"Lumberjack input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::CircuitBreaker::HalfOpenBreaker, :level=>:warn}
{:timestamp=>"2015-11-04T12:12:33.829000+0000", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Lumberjack input", :exception=>LogStash::SizedQueueTimeout::TimeoutError, :level=>:warn}
{:timestamp=>"2015-11-04T12:12:33.839000+0000", :message=>"Lumberjack input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::CircuitBreaker::HalfOpenBreaker, :level=>:warn}
{:timestamp=>"2015-11-04T12:12:43.596000+0000", :message=>"SIGTERM received. Shutting down the pipeline.", :level=>:warn}
{:timestamp=>"2015-11-04T12:12:48.608000+0000", :message=>["INFLIGHT_EVENTS_REPORT", "2015-11-04T12:12:48+00:00", {"input_to_filter"=>20, "filter_to_output"=>0, "outputs"=>[]}], :level=>:warn}
{:timestamp=>"2015-11-04T12:12:53.345000+0000", :message=>"SIGTERM received. Shutting down the pipeline.", :level=>:warn}
Logstash does not seem to be able to extract values from the 'file' field when used with the Lumberjack input plugin / Logstash Forwarder (does work if extracting from the path field when the input is file). When I comment out the grok section, Logstash works properly and you can see the file field in the output below:
"message" => "\tat org.apache.activemq.util.ServiceSupport.stop(ServiceSupport.java:71)[185:org.apache.activemq.activemq-osgi:5.11.0.redhat-620133]",
"@version" => "1",
"@timestamp" => "2015-11-04T12:22:05.829Z",
"file" => "/var/log/karaf/event-broker-container-abc-fb99.log.1",
"host" => "abc-fb99.abc.dev",
"offset" => "541719",
"type" => "karaf"
Is this a bug or how is it possible to Grok on the file field?
Thanks,