# Logstash Does Not Read In Last Line of Log

**URL:** https://discuss.elastic.co/t/logstash-does-not-read-in-last-line-of-log/2706
**Category:** Logstash
**Created:** [June 15, 2015, 2:43pm UTC](https://discuss.elastic.co/t/logstash-does-not-read-in-last-line-of-log/2706 "2015-06-15T14:43:31Z")
**Posts on this page:** 17
**Page:** 1

<div class="post-metadata">

### Author: ![michaellizhou](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michaellizhou/32/4143_2.png) [@michaellizhou](https://discuss.elastic.co/u/michaellizhou)
#### Post date: [June 15, 2015, 2:43pm UTC](https://discuss.elastic.co/t/logstash-does-not-read-in-last-line-of-log/2706/1 "2015-06-15T14:43:31Z")

</div>

So recently noticed that my logstash config is not reading in the last line from my log file. I think that it may be my input config fault though I do not see any problems with it right now. What else could it be? Here is my input config:  
input {

> file {  
> type =\> "java"  
> codec =\> multiline {  
> pattern =\> "^\s"  
> what =\> "previous"  
> }  
> path =\> "something"  
> start\_position =\> "beginning"  
> sincedb\_path =\> "somewhere"  
> }  
> file {  
> type =\> "communication"  
> codec =\> multiline {  
> pattern =\> "cool pattern"  
> negate =\> "true"  
> what =\> "previous"  
> }  
> path =\> "somwhereelse"  
> start\_position =\> "beginning"  
> sincedb\_path =\> "somwhereelse"  
> }

> file {  
> type =\> "sometype"  
> codec =\> multiline {  
> pattern =\> "coolpattern"  
> negate =\> "true"  
> what =\> "previous"  
> }  
> path =\> "somwhereelse"  
> start\_position =\> "beginning"  
> sincedb\_path =\> "somwhereelse"  
> }

> }

After much research fixed it. Had to read a lot of how they personally attacked the problem. But if someone is looking for solution just reply back.

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [June 16, 2015, 12:18am UTC](https://discuss.elastic.co/t/logstash-does-not-read-in-last-line-of-log/2706/2 "2015-06-16T00:18:15Z")

</div>

Providing your solution would help others!

---

<div class="post-metadata">

### Author: ![michaellizhou](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michaellizhou/32/4143_2.png) [@michaellizhou](https://discuss.elastic.co/u/michaellizhou)
#### Post date: [June 16, 2015, 12:45pm UTC](https://discuss.elastic.co/t/logstash-does-not-read-in-last-line-of-log/2706/3 "2015-06-16T12:45:21Z")

</div>

Its just a really simple fix but I could not find it in I guess modern forms/community. Had to dig deep into other chats. So basically you move the multiline command to the filter rather then the codec that is used in the input. You should notice that in the 1.5.0 release that multiline is placed in filter, but I read somewhere that people were using multiline in input as a codec. Bad mistake...  
(Problem) Multiline is waiting for the last line but cannot determine the last line is indeed the last line so it has to wait for another log to be read in. Problem real time errors will not show up...slow logs...more bad stuff  
(Fix) So now there is an update such that there will be a time out on the last log. Now you will never miss your last log. But you have to make sure that you MOVE multiline into FILTER and not input. I tried leaving the command within input block and problem persists. GL to everyone else learning like me.

---

<div class="post-metadata">

### Author: ![chaitanyavvs](https://avatars.discourse-cdn.com/v4/letter/c/46a35a/32.png) [@chaitanyavvs](https://discuss.elastic.co/u/chaitanyavvs)
#### Post date: [June 18, 2015, 8:28pm UTC](https://discuss.elastic.co/t/logstash-does-not-read-in-last-line-of-log/2706/4 "2015-06-18T20:28:21Z")

</div>

Apologies @michaellizhou for opening an old thread. But i have config where in i have the multiline as a filter and still the last line of the log is not considered.

> input{  
> stdin {  
> }  
> }

> filter {

> multiline {  
> pattern =\> "^%{DATE}"  
> negate =\> true  
> what =\> previous  
> }

> grok {  
> match =\>  
> [  
> "message","(?%{DATE}%{SPACE}(?%{HOUR}:%{MINUTE}))-(?%{WORD}-%{LOGLEVEL:loggingLevel}):%{DATA}Run:%{DATA}%{NUMBER:runNo:int},%{DATA}ContextKey:%{DATA}%{NUMBER:ctx:int},%{DATA}COBDate:%{DATA}(?%{MONTHDAY}-%{MONTH}-%{YEAR}),%{DATA}Step:%{GREEDYDATA:step},%{DATA}Type:%{GREEDYDATA:runType},%{DATA}Status:%{DATA}%{WORD:status},%{DATA}Time:%{DATA}%{NUMBER:timeTaken:int}%{GREEDYDATA}",  
> "message","(?%{DATE}%{SPACE}(?%{HOUR}:%{MINUTE}))-(?%{WORD}-%{LOGLEVEL:loggingLevel}):%{DATA}Run:%{DATA}%{NUMBER:runNo:int},%{DATA}ContextKey:%{DATA}%{NUMBER:ctx:int},%{DATA}COBDate:%{DATA}(?%{MONTHDAY}-%{MONTH}-%{YEAR}),%{DATA}Step:%{GREEDYDATA:step},%{DATA}Type:%{GREEDYDATA:runType},%{DATA}Status:%{DATA}%{WORD:status}%{GREEDYDATA}",  
> "message","(?%{DATE}%{SPACE}(?%{HOUR}:%{MINUTE}))-(?%{WORD}-%{LOGLEVEL:loggingLevel}):%{GREEDYDATA}"  
> ]

> }

> date {  
> match =\> ["loggedtime","MM/dd/yyyy HH:mm"]  
> }

> }

> output {  
> stdout {  
> codec=\>rubydebug  
> }  
> }

and the sample data is :

```
06/16/15 11:30-XX-INFO: Run:1,ContextKey: 1, COBDate: 31-May-2015, Step: X, Type: Whole,Status:Running
06/16/15 11:40-XX-INFO: Run:1, ContextKey: 1, COBDate: 31-May-2015, Step: X 1, Type: Individual, Status: Running

```

Request your help on this

---

<div class="post-metadata">

### Author: ![michaellizhou](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michaellizhou/32/4143_2.png) [@michaellizhou](https://discuss.elastic.co/u/michaellizhou)
#### Post date: [June 18, 2015, 9:40pm UTC](https://discuss.elastic.co/t/logstash-does-not-read-in-last-line-of-log/2706/5 "2015-06-18T21:40:51Z")

</div>

Quick question what version of logstash are you use?

---

<div class="post-metadata">

### Author: ![chaitanyavvs](https://avatars.discourse-cdn.com/v4/letter/c/46a35a/32.png) [@chaitanyavvs](https://discuss.elastic.co/u/chaitanyavvs)
#### Post date: [June 19, 2015, 1:41pm UTC](https://discuss.elastic.co/t/logstash-does-not-read-in-last-line-of-log/2706/6 "2015-06-19T13:41:52Z")

</div>

I am using logstash 1.5.0 , Apologies for missing that.

---

<div class="post-metadata">

### Author: ![michaellizhou](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michaellizhou/32/4143_2.png) [@michaellizhou](https://discuss.elastic.co/u/michaellizhou)
#### Post date: [June 19, 2015, 1:52pm UTC](https://discuss.elastic.co/t/logstash-does-not-read-in-last-line-of-log/2706/7 "2015-06-19T13:52:32Z")

</div>

That is just really odd because I know they patched up the last log issue. Can you paste a sample of what you are getting for an output? I quickly threw your config into logstash and I am getting an error.

```
The error reported is: 
  undefined group option: /(?(?:(?:(?:(?:0?[1-9]|1[0-2]))[\/-](?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))[\/-](?:(?>\d\d){1,2}))|(?:(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))[.\/-](?:(?:0?[1-9]|1[0-2]))[.\/-](?:(?>\d\d){1,2})))(?:\s*)(?(?:(?:2[0123]|[01]?[0-9])):(?:(?:[0-5][0-9]))))-(?(?:\b\w+\b)-(?<LOGLEVEL:loggingLevel>([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?))):(?:.*?)Run:(?:.*?)(?<NUMBER:runNo:int>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))))),(?:.*?)ContextKey:(?:.*?)(?<NUMBER:ctx:int>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))))),(?:.*?)COBDate:(?:.*?)(?(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))-(?:\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b)-(?:(?>\d\d){1,2})),(?:.*?)Step:(?<GREEDYDATA:step>.*),(?:.*?)Type:(?<GREEDYDATA:runType>.*),(?:.*?)Status:(?:.*?)(?<WORD:status>\b\w+\b),(?:.*?)Time:(?:.*?)(?<NUMBER:timeTaken:int>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:.*)/m

```

So what ever that means its not liking it when I try on my logstash. Maybe try to upload your config on github or anything else that is easier to copy and paste the code.

---

<div class="post-metadata">

### Author: ![chaitanyavvs](https://avatars.discourse-cdn.com/v4/letter/c/46a35a/32.png) [@chaitanyavvs](https://discuss.elastic.co/u/chaitanyavvs)
#### Post date: [June 19, 2015, 2:20pm UTC](https://discuss.elastic.co/t/logstash-does-not-read-in-last-line-of-log/2706/8 "2015-06-19T14:20:47Z")

</div>

Thanks a lot for taking your time and trying it out. You Could keep this small GROK pattern, instead of the list of the patterns i had.

> input {  
> stdin {  
> }  
> }  
> grok {  
> match =\>  
> [  
> "message","(?%{DATE}%{SPACE}(?%{HOUR}:%{MINUTE}))-(?%{WORD}-%{LOGLEVEL:loggingLevel}):%{GREEDYDATA}"  
> ]

> }

> output {  
> stdout {  
> codec=\>rubydebug  
> }  
> }

I have specified the sample data in the first request itself. Please do let me know if you need any more information from my side.

---

<div class="post-metadata">

### Author: ![michaellizhou](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michaellizhou/32/4143_2.png) [@michaellizhou](https://discuss.elastic.co/u/michaellizhou)
#### Post date: [June 19, 2015, 3:20pm UTC](https://discuss.elastic.co/t/logstash-does-not-read-in-last-line-of-log/2706/9 "2015-06-19T15:20:03Z")

</div>

Same problem btw your missing the filter part in the config you just pasted. I do not know if your config is compiling for you or not but you should check with --configtest and then try to actually run it. I can say its definitely not compiling on my end. Must be something up with your grok pattern. Since it seems like you are not that familiar with grok patterns I would recommend doing this:  
Write the pattern yourself! it might be time consuming at first but you learn a lot about regex, but once you figure this out then start using some of the default patterns that is provided by grok. That is at least what I did. What your config gives me

 ![](https://us1.discourse-cdn.com/elastic/original/2X/0/035c68589d9fbf93de93481249c8c2ea8a8aeaf8.png)  
here is quick start to your date parsing. you know what to do from here

```
"message" , "(?<date>\d+\/\d+\/\d+s*\d+:\d+)...
```

---

<div class="post-metadata">

### Author: ![chaitanyavvs](https://avatars.discourse-cdn.com/v4/letter/c/46a35a/32.png) [@chaitanyavvs](https://discuss.elastic.co/u/chaitanyavvs)
#### Post date: [June 19, 2015, 5:11pm UTC](https://discuss.elastic.co/t/logstash-does-not-read-in-last-line-of-log/2706/10 "2015-06-19T17:11:18Z")

</div>

Thanks again for your time @michaellizhou . One thing, is there is a small error in the configuration which i sent. Its a copy-paste issue. I just wanted to let you know that i am working on Windows platform and below is the configtest result

> C:\Users\o558781\Desktop\>logstash.bat -f "C:\Users\o558781\Desktop\logstashconf.conf" --configtest  
> io/console not supported; tty will not be manipulated  
> Configuration OK

Below is the config file i have used

> C:\Users\o558781\Desktop\>type logstashconf.conf

> ```
> input {
> stdin {
> }
> }
> 
> ```
> 
> filter {  
> grok {  
> match =\>  
> [  
> "message","(?%{DATE}%{SPACE}(?%{HOUR}:%{MINUTE}))-(?%{WORD}-%{LOGLEVEL:loggingLevel}):%{GREEDYDATA}"  
> ]  
> }  
> }  
> output {  
> stdout {  
> codec=\>rubydebug  
> }  
> }

Please do let me know if you need any more information from my side

---

<div class="post-metadata">

### Author: ![linrongfu26](https://avatars.discourse-cdn.com/v4/letter/l/d07c76/32.png) [@linrongfu26](https://discuss.elastic.co/u/linrongfu26)
#### Post date: [April 1, 2016, 9:45am UTC](https://discuss.elastic.co/t/logstash-does-not-read-in-last-line-of-log/2706/11 "2016-04-01T09:45:30Z")

</div>

i have the same problem and don't know how to fix it.  
my log file has 6 line ,but the stdout only print 5 line of it, after i add a new line to the log file , the stdout print the 6 line.

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [April 1, 2016, 8:50pm UTC](https://discuss.elastic.co/t/logstash-does-not-read-in-last-line-of-log/2706/12 "2016-04-01T20:50:14Z")

</div>

Might be be better to start a new thread, this one is 9 months old 🙂

---

<div class="post-metadata">

### Author: ![gilisade](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gilisade/32/98740_2.png) [@gilisade](https://discuss.elastic.co/u/gilisade)
#### Post date: [June 26, 2016, 1:44pm UTC](https://discuss.elastic.co/t/logstash-does-not-read-in-last-line-of-log/2706/13 "2016-06-26T13:44:01Z")

</div>

great solution ,  
had the same problem , messages were not commimg in on-line

DO NOT USE MULTILINE AS AN INPUT CODEC!!!

---

<div class="post-metadata">

### Author: ![sumitkgaur](https://avatars.discourse-cdn.com/v4/letter/s/f19dbf/32.png) [@sumitkgaur](https://discuss.elastic.co/u/sumitkgaur)
#### Post date: [September 7, 2016, 8:18am UTC](https://discuss.elastic.co/t/logstash-does-not-read-in-last-line-of-log/2706/14 "2016-09-07T08:18:27Z")

</div>

Multiline filter has been deprecated in favor of multiline-codec But the problem still persist with codec .  
Does anybody has any workaround ? Last line of the logs is not getting processed

input {  
beats {  
port =\> 5044  
codec =\> multiline {  
pattern =\> "^\s"  
negate =\> "true"  
what =\> "previous"  
}  
}  
}

---

<div class="post-metadata">

### Author: ![krishna\_chaitanya](https://avatars.discourse-cdn.com/v4/letter/k/b5a626/32.png) [@krishna\_chaitanya](https://discuss.elastic.co/u/krishna_chaitanya)
#### Post date: [February 10, 2017, 4:03pm UTC](https://discuss.elastic.co/t/logstash-does-not-read-in-last-line-of-log/2706/15 "2017-02-10T16:03:47Z")

</div>

Is there any solution to this with multiline codec? I am using logstash 5.2 and having the same issue. Unable to read the last line.

---

<div class="post-metadata">

### Author: ![vdimitrova](https://avatars.discourse-cdn.com/v4/letter/v/9dc877/32.png) [@vdimitrova](https://discuss.elastic.co/u/vdimitrova)
#### Post date: [May 26, 2017, 10:21am UTC](https://discuss.elastic.co/t/logstash-does-not-read-in-last-line-of-log/2706/16 "2017-05-26T10:21:24Z")

</div>

I used the following configuration of the codec multiline and it appears to work correctly. Just set :

**auto\_flush\_interval =\> 1**

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 4:26am UTC](https://discuss.elastic.co/t/logstash-does-not-read-in-last-line-of-log/2706/17 "2017-07-06T04:26:19Z")

</div>