# Logstash: Log records fail to load in large log file but work when run separately

**URL:** <https://discuss.elastic.co/t/logstash-log-records-fail-to-load-in-large-log-file-but-work-when-run-separately/69384>\
**Category:** Logstash\
**Created:** [December 19, 2016, 4:29am UTC](https://discuss.elastic.co/t/logstash-log-records-fail-to-load-in-large-log-file-but-work-when-run-separately/69384 "2016-12-19T04:29:25Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Matt\_Simcox](https://avatars.discourse-cdn.com/v4/letter/m/c77e96/32.png) [@Matt\_Simcox](https://discuss.elastic.co/u/Matt_Simcox)\
**Post date:** [December 19, 2016, 4:29am UTC](https://discuss.elastic.co/t/logstash-log-records-fail-to-load-in-large-log-file-but-work-when-run-separately/69384/1 "2016-12-19T04:29:25Z")

</div>

Filebeat is feeding data into logstash and passing this onto ES. I'm monitoring the load of around 5 million records and notice that I'm getting a lot of \_grokparsefailure.

when I drill into the logstash output I can see the following:

15:11:09.999 [[main]\>worker1] WARN logstash.filters.date - Failed parsing date from field {:field=\>"time", :value=\>"2016-10-02\_02:59:40.997", :exception=\>"Invalid format: "2016-10-02\_02:59:40.997" is malformed at "\_02:59:40.997"", :config\_parsers=\>"yyyy-MM-dd\_HH:mm:ss.SSS,yyyy-MM-dd HH:mm:ss.SSS", :config\_locale=\>"default=en\_AU"}

The record that caused this to fail is:

2016-10-02\_02:59:40.997 [transaction-1] INFO companyA.engine.TransactionHandler - Transaction Completed : SOCKETID=62437,TXNREFERENCE=,CLIENTID=10000031,RESPONSECODE=-1,RESPONSETEXT=TRANSACTION NOT FOUND,DURATION=0,TRANSACTIONTYPE=STATUS[STATUS],INTERFACE=CREDITCARD

I tested this in GrockDebugger and it worked fine.  
I loaded it separately in it's own log file and it loaded fine.

The logstash pipeline conf file is:

input {

# stdin {

# }

beats {  
port =\> "5043"  
}  
}  
filter {  
grok {  
patterns\_dir =\> ["./patterns"]  
match =\> { "message" =\> "%{TRANTRACK\_DATE:time} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} - %{NOTSPACE} Completed :\s\w\*=(%{WORD:socketval})?,\w\*=(%{WORD:txnref})?,\w\*=(%{WORD:clientid})?,\w\*=(%{NUMBER:respcode})?,\w\*=(%{MULTI\_WORD:resptext})?,\w\*=(%{WORD:duration})?,\w\*=(%{TRAN\_WITH\_SUBTYPE:txntype})?,\w\*=(%{WORD:interface})?" }  
overwrite =\> ["message"]   
}  
if ([message] =~ "Transaction Start") {  
drop {}  
}  
date {  
match =\> ["time","yyyy-MM-dd\_HH:mm:ss.SSS","yyyy-MM-dd HH:mm:ss.SSS"]  
#timezone =\> "Australia/Sydney"  
}  
}

output {  
#stdout {codec =\> json}  
elasticsearch {  
hosts =\> ["x.x.x.x:9200"]  
index =\> "companyAtrantrack-companyB-%{+YYYY.MM.dd}"  
document\_type =\> "%{[@metadata][type]}"  
}  
}

Each record that failed as part of the main load has worked in a smaller file. I'm confused.

---

<div class="post-metadata">

**Author:** ![jakommo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jakommo/32/6716_2.png) [@jakommo](https://discuss.elastic.co/u/jakommo)\
**Post date:** [December 20, 2016, 3:28pm UTC](https://discuss.elastic.co/t/logstash-log-records-fail-to-load-in-large-log-file-but-work-when-run-separately/69384/2 "2016-12-20T15:28:54Z")

</div>

The pattern seems ok and it works for me with just using 2016-10-02\_02:59:40.997.  
Might be worth testing this again with the original file, maybe there are (invisible) control chars in there or something.  
The issue seems to be with this part "02:59:40.997"

---

<div class="post-metadata">

**Author:** ![jakommo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jakommo/32/6716_2.png) [@jakommo](https://discuss.elastic.co/u/jakommo)\
**Post date:** [December 20, 2016, 5:39pm UTC](https://discuss.elastic.co/t/logstash-log-records-fail-to-load-in-large-log-file-but-work-when-run-separately/69384/3 "2016-12-20T17:39:11Z")

</div>

Ok, just saw your [other post](https://discuss.elastic.co/t/logstash-fails-to-load-indian-log-records-due-to-daylight-savings-change-on-host-machine/69525) and I think you already answered your question there.

After commenting in the timezone =\> "Australia/Sydney" (used UTC+1 before) I'm also getting this error and it's because there is no 02:00-03:00 on 2016-10-02 (daylight savings switch over).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 17, 2017, 5:39pm UTC](https://discuss.elastic.co/t/logstash-log-records-fail-to-load-in-large-log-file-but-work-when-run-separately/69384/4 "2017-01-17T17:39:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.