# Logstash stalls with grok pattern with optional match
**URL:** https://discuss.elastic.co/t/logstash-stalls-with-grok-pattern-with-optional-match/47215
**Category:** Logstash
**Created:** [April 13, 2016, 8:30am UTC](https://discuss.elastic.co/t/logstash-stalls-with-grok-pattern-with-optional-match/47215 "2016-04-13T08:30:46Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@kerk1v](https://discuss.elastic.co/u/kerk1v)
#### Post date: [April 13, 2016, 8:30am UTC](https://discuss.elastic.co/t/logstash-stalls-with-grok-pattern-with-optional-match/47215/1 "2016-04-13T08:30:46Z")
Hi there,
Got the following error when trying to match an optional part in the logs. Seems related to Ruby, here is the full log I get when i hit Ctrl+c when Logstash stalls and starts taking every bit of CPU on my 4-CPU machine:
```
{"inflight_count"=>375, "stalling_thread_info"=>{["LogStash::Filters::Grok", {"match"=>{"message"=>" \\- \\- \\[%{HTTPDATE:timestamp}\\] Host:\"%{HOSTNAME}\" RequestedHost:\"%{HOSTNAME}\" \"Requested:\" \"%{NOTSPACE} %{UNIXPATH:full_path}(%{URIPARAM:parameters})? %{NOTSPACE} %{INT:result_code} %{INT:bytes}"}}]=>[{"thread_id"=>20, "name"=>"[main]>worker1", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb:177:in `match'"}, {"thread_id"=>21, "name"=>"[main]>worker2", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb:177:in `match'"}, {"thread_id"=>22, "name"=>"[main]>worker3", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb:177:in `match'"}]}} {:level=>:warn}
{"inflight_count"=>375, "stalling_thread_info"=>{["LogStash::Filters::Grok", {"match"=>{"message"=>" \\- \\- \\[%{HTTPDATE:timestamp}\\] Host:\"%{HOSTNAME}\" RequestedHost:\"%{HOSTNAME}\" \"Requested:\" \"%{NOTSPACE} %{UNIXPATH:full_path}(%{URIPARAM:parameters})? %{NOTSPACE} %{INT:result_code} %{INT:bytes}"}}]=>[{"thread_id"=>20, "name"=>"[main]>worker1", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb:177:in `match'"}, {"thread_id"=>21, "name"=>"[main]>worker2", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb:177:in `match'"}, {"thread_id"=>22, "name"=>"[main]>worker3", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb:177:in `match'"}]}} {:level=>:warn}
{"inflight_count"=>375, "stalling_thread_info"=>{["LogStash::Filters::Grok", {"match"=>{"message"=>" \\- \\- \\[%{HTTPDATE:timestamp}\\] Host:\"%{HOSTNAME}\" RequestedHost:\"%{HOSTNAME}\" \"Requested:\" \"%{NOTSPACE} %{UNIXPATH:full_path}(%{URIPARAM:parameters})? %{NOTSPACE} %{INT:result_code} %{INT:bytes}"}}]=>[{"thread_id"=>20, "name"=>"[main]>worker1", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb:177:in `match'"}, {"thread_id"=>21, "name"=>"[main]>worker2", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb:177:in `match'"}, {"thread_id"=>22, "name"=>"[main]>worker3", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb:177:in `match'"}]}} {:level=>:warn}
{"inflight_count"=>375, "stalling_thread_info"=>{["LogStash::Filters::Grok", {"match"=>{"message"=>" \\- \\- \\[%{HTTPDATE:timestamp}\\] Host:\"%{HOSTNAME}\" RequestedHost:\"%{HOSTNAME}\" \"Requested:\" \"%{NOTSPACE} %{UNIXPATH:full_path}(%{URIPARAM:parameters})? %{NOTSPACE} %{INT:result_code} %{INT:bytes}"}}]=>[{"thread_id"=>20, "name"=>"[main]>worker1", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb:177:in `match'"}, {"thread_id"=>21, "name"=>"[main]>worker2", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb:177:in `match'"}, {"thread_id"=>22, "name"=>"[main]>worker3", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb:177:in `match'"}]}} {:level=>:warn}
{"inflight_count"=>375, "stalling_thread_info"=>{["LogStash::Filters::Grok", {"match"=>{"message"=>" \\- \\- \\[%{HTTPDATE:timestamp}\\] Host:\"%{HOSTNAME}\" RequestedHost:\"%{HOSTNAME}\" \"Requested:\" \"%{NOTSPACE} %{UNIXPATH:full_path}(%{URIPARAM:parameters})? %{NOTSPACE} %{INT:result_code} %{INT:bytes}"}}]=>[{"thread_id"=>20, "name"=>"[main]>worker1", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb:177:in `match'"}, {"thread_id"=>21, "name"=>"[main]>worker2", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb:177:in `match'"}, {"thread_id"=>22, "name"=>"[main]>worker3", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb:177:in `match'"}]}} {:level=>:warn}
{"inflight_count"=>375, "stalling_thread_info"=>{["LogStash::Filters::Grok", {"match"=>{"message"=>" \\- \\- \\[%{HTTPDATE:timestamp}\\] Host:\"%{HOSTNAME}\" RequestedHost:\"%{HOSTNAME}\" \"Requested:\" \"%{NOTSPACE} %{UNIXPATH:full_path}(%{URIPARAM:parameters})? %{NOTSPACE} %{INT:result_code} %{INT:bytes}"}}]=>[{"thread_id"=>20, "name"=>"[main]>worker1", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb:177:in `match'"}, {"thread_id"=>21, "name"=>"[main]>worker2", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb:177:in `match'"}, {"thread_id"=>22, "name"=>"[main]>worker3", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb:177:in `match'"}]}} {:level=>:warn}
```
Any idea?
---
### Author:  [@michbsd](https://discuss.elastic.co/u/michbsd)
#### Post date: [June 7, 2016, 4:34pm UTC](https://discuss.elastic.co/t/logstash-stalls-with-grok-pattern-with-optional-match/47215/2 "2016-06-07T16:34:48Z")
Hi,
I am seeing this as well.. Did you manage to solve it ?
---
### Author:  [@sambroj](https://discuss.elastic.co/u/sambroj)
#### Post date: [June 10, 2016, 11:05am UTC](https://discuss.elastic.co/t/logstash-stalls-with-grok-pattern-with-optional-match/47215/3 "2016-06-10T11:05:22Z")
Hi,
I am also experiencing this problem with the grok filter. Previously, LS was able to ingest my logs without any problems.
My version is logstash-2.3.2-1.noarch.
Could any developer take a look at it?
---
### Author:  [@purbon](https://discuss.elastic.co/u/purbon)
#### Post date: [June 10, 2016, 2:25pm UTC](https://discuss.elastic.co/t/logstash-stalls-with-grok-pattern-with-optional-match/47215/4 "2016-06-10T14:25:13Z")
Can you share your configuration and sample log lines? this looks like an stalled pipeline is preventing you to move forward.
---
### Author:  [@sambroj](https://discuss.elastic.co/u/sambroj)
#### Post date: [June 13, 2016, 8:18am UTC](https://discuss.elastic.co/t/logstash-stalls-with-grok-pattern-with-optional-match/47215/5 "2016-06-13T08:18:48Z")
This problem occurs when the number of lines inside the logfile you want to analyze is big. Debugging with a logfile containing 10 lines works and the grok filter evaluates the lines perfectly. It is not a problem with the regular expression patterns and the very same patterns worked before.
Increasing the number of lines makes logstash to stall by the grok filter:
{"inflight\_count"=\>45, "stalling\_thread\_info"=\>{["LogStash::Filters::Grok", {"patterns\_dir"=\>["/etc/logstash/patterns/"], "match"=\>{"message"=\>"%{TRAN
SFER\_CLASSIC}"}, "periodic\_flush"=\>"true", "remove\_field"=\>["message"]}]=\>[{"thread\_id"=\>20, "name"=\>"[main]\>worker1", "current\_call"=\>"[...]/vendor/b
undle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb:177:in `match'"}]}} {:level=\>:warn}
I have run logstash on the same machine where the logs are located trying to isolate the problem. I have simplified the configuration file and the problem still persists:
input {
file {
path =\> "/var/test/\*.log"
start\_position =\> beginning
sincedb\_path =\> "/dev/null"
type =\> "billing"
ignore\_older =\> 0
}
}
filter {
grok {
patterns\_dir =\> ["/etc/logstash/patterns/"]
match =\> { "message" =\> "%{TRANSFER\_CLASSIC}" }
periodic\_flush =\> true
remove\_field =\> ["message"]
}
date {
match =\> ["billing\_time", "MM.dd HH:mm:ss"]
timezone =\> "CET"
remove\_field =\> ["billing\_time"]
}
}
output {
stdout {
codec =\> rubydebug
}
}
I want to remark that the patterns have been tested and worked with previous versions of logstash. Version is logstash-2.3.2-1.noarch. Operating system is Scientific Linux 7 (based on RedHat) and we use openjdk.
Any help will be appreciated.
---
### Author:  [@michbsd](https://discuss.elastic.co/u/michbsd)
#### Post date: [July 4, 2016, 1:53pm UTC](https://discuss.elastic.co/t/logstash-stalls-with-grok-pattern-with-optional-match/47215/6 "2016-07-04T13:53:34Z")
it is most likely due to the grok pattern choking on something.
I ended writing up a manual parser in ruby (plenty of examples available on the interwebs) and parsing the problematic log file manually.
It showed that URLs with "app-android://" choked the parsing process - and I changed the field type to something less strict (like NOTSPACE) and the problem went away..
Mike
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 4:49am UTC](https://discuss.elastic.co/t/logstash-stalls-with-grok-pattern-with-optional-match/47215/7 "2017-07-06T04:49:44Z")