# Grokparsefailure with NXlog and Logstash

**URL:** <https://discuss.elastic.co/t/grokparsefailure-with-nxlog-and-logstash/36210>\
**Category:** Logstash\
**Created:** [December 2, 2015, 5:37pm UTC](https://discuss.elastic.co/t/grokparsefailure-with-nxlog-and-logstash/36210 "2015-12-02T17:37:58Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sandeep\_Krishnankutt](https://avatars.discourse-cdn.com/v4/letter/s/258eb7/32.png) [@Sandeep\_Krishnankutt](https://discuss.elastic.co/u/Sandeep_Krishnankutt)\
**Post date:** [December 2, 2015, 5:37pm UTC](https://discuss.elastic.co/t/grokparsefailure-with-nxlog-and-logstash/36210/1 "2015-12-02T17:37:59Z")

</div>

I seem to have a strange(or not) issue with logstash where I am seeing random grokparsefailure.

I am using NXLog to ship IIS logs from 5 of my webservers to Logstash. The NXlog config is as below

> Module xm\_json

> Module xm\_fileop

> Module im\_file File "D:\IISLogs\W3SVC2\u\_ex\*.log" PollInterval 120 DirCheckInterval 120 ReadFromLast True SavePos True Exec if $raw\_event =~ /^#/ drop();

> Module om\_tcp Host \<\> Port 5544 OutputType LineBased

> Path iis\_core =\> out\_logstash

The webservers have a very high number of requests being served per second and hence a large number of request is being send to logstash from the 5 webservers. I can see that I am having some log entries marked as 'grokparsefailure' and seem to be very random.

Logstash conf

> input {  
> tcp {  
> port =\> 5544  
> }  
> }

> filter {
> 
> #ignore log comments  
> if [message] =~ "^#" {  
> drop {}  
> }

> grok {  
> # check that fields match your IIS log settings  
> match =\> ["message", "%{TIMESTAMP\_ISO8601:log\_timestamp} %{WORD:iisSite} %{HOSTNAME:sourcehost} %{IPORHOST:hostip} %{WORD:method} %{URIPATH:page} %{NOTSPACE:querystring} %{WORD:port} %{NOTSPACE:username} %{IPORHOST:clientip} %{NOTSPACE:version} %{NOTSPACE:useragent} %{NOTSPACE:cscookie} %{NOTSPACE:referer} %{NOTSPACE:cshost} %{NUMBER:scstatus:int} %{NUMBER:subresponse} %{NUMBER:win32substatus:int} %{NUMBER:scbytes:int} %{NUMBER:csbytes:int} %{NUMBER:timetaken:int}"]  
> }
> 
> date {  
> match =\> ["log\_timestamp", "YYYY-MM-dd HH:mm:ss"]  
> timezone =\> "Etc/UCT"  
> }
> 
> useragent {  
> source=\> "useragent"  
> prefix=\> "browser"  
> }
> 
> mutate {  
> remove\_field =\> ["host"]  
> remove\_field =\> ["message"]  
> remove\_field =\> ["cscookie"]  
> }  
> }

> output {  
> stdout { codec =\> rubydebug }  
> if "\_grokparsefailure" in [tags] {  
> file { path =\> "C:\logstash\log\failed\_grok\_events-%{+YYYY-MM-dd}" }  
> }  
> elasticsearch {  
> hosts =\> "127.0.0.1:9200"  
> index =\> "logstash-%{+YYYY.MM.dd}"  
> }  
> }

From the failed grok events, I am able to see this

> {"message":"\r","@version":"1","@timestamp":"2015-11-29T21:31:26.518Z","tags":["\_grokparsefailure"]}

The message part just has '\r' in it and I cant find a line in IIS log that would correspond to it. (I have tested the implementation with a bad grok pattern and the message is showing the corresponding iis log).

Does anyone know if this is NXLog issue or Logstash issue or something that happens when a large number of requests hits logstash.?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 3, 2015, 7:29am UTC](https://discuss.elastic.co/t/grokparsefailure-with-nxlog-and-logstash/36210/2 "2015-12-03T07:29:08Z")

</div>

\r is the carriage return character, i.e. one of the two characters in the Windows line break. Are there any blank lines in the input log? The codec strips trailing newline characters (\n) but perhaps not carriage returns so I suspect that blank lines in Windows files show up as \r in Logstash.

---

<div class="post-metadata">

**Author:** ![Sandeep\_Krishnankutt](https://avatars.discourse-cdn.com/v4/letter/s/258eb7/32.png) [@Sandeep\_Krishnankutt](https://discuss.elastic.co/u/Sandeep_Krishnankutt)\
**Post date:** [December 3, 2015, 4:47pm UTC](https://discuss.elastic.co/t/grokparsefailure-with-nxlog-and-logstash/36210/3 "2015-12-03T16:47:02Z")

</div>

Thank you for the reply magnus. I did a search using a script and cant see any blank lines. As these are just plain IISLogs, I dont think there would be any blanks lines in the logs. its so strange that it still shows up though. I have a similar setup for our QA environment and cant see any grokparsefailure in there.

I wonder if it could be a race condition where IIS puts in a new line and nxlog sends the log before it is updated or could be that logstash in some way fails when it receives a large number of log entries.?

PS: I have read many topics that you have replied to and I feel you are doing a great job. Thanks a lot.

---

<div class="post-metadata">

**Author:** ![xtruthx](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xtruthx/32/10435_2.png) [@xtruthx](https://discuss.elastic.co/u/xtruthx)\
**Post date:** [June 21, 2016, 1:36pm UTC](https://discuss.elastic.co/t/grokparsefailure-with-nxlog-and-logstash/36210/4 "2016-06-21T13:36:34Z")

</div>

plz can you tell me how you solve the problem?

thx

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:51am UTC](https://discuss.elastic.co/t/grokparsefailure-with-nxlog-and-logstash/36210/5 "2017-07-06T04:51:29Z")

</div>