# Can not parse the whole file but stopped at 1 am(logstash 2.4)

**URL:** <https://discuss.elastic.co/t/can-not-parse-the-whole-file-but-stopped-at-1-am-logstash-2-4/62542>\
**Category:** Logstash\
**Created:** [October 8, 2016, 4:27am UTC](https://discuss.elastic.co/t/can-not-parse-the-whole-file-but-stopped-at-1-am-logstash-2-4/62542 "2016-10-08T04:27:24Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![duwupeng](https://avatars.discourse-cdn.com/v4/letter/d/65b543/32.png) [@duwupeng](https://discuss.elastic.co/u/duwupeng)\
**Post date:** [October 8, 2016, 4:27am UTC](https://discuss.elastic.co/t/can-not-parse-the-whole-file-but-stopped-at-1-am-logstash-2-4/62542/1 "2016-10-08T04:27:24Z")

</div>

The following config can run smoothly  
filter {  
if(" - response:" in [message]){  
grok {  
match =\> { "message" =\> "%{GREEDYDATA:timestamp} %{GREEDYDATA:thread} INFO - response:%{GREEDYDATA:serviceName} %{GREEDYDATA:serviceMethod} %{GREEDYDATA:sessionId} %{GREEDYDATA:ipAddress} %{GREEDYDATA:callerFrom} %{GREEDYDATA:operatorId} %{GREEDYDATA:operat orName} %{GREEDYDATA:customerId} %{GREEDYDATA:customerName} %{NUMBER:timeConsumed}ms"}  
}  
date {  
match =\> ["timestamp", "MM-dd HH:mm:ss SSS"]  
}  
mutate { remove\_field =\> "thread" }  
mutate { add\_field =\> { log\_type =\> "ksresponse" } }  
} else if(" - SQL Preparing: " in [message]){  
grok {  
match =\> { "message" =\> "%{GREEDYDATA:timestamp} %{GREEDYDATA:thread} DEBUG - SQL Preparing: %{GREEDYDATA:sql} args:%{GREEDYDATA:sqlArgs}"}  
}  
date {  
match =\> ["timestamp", "MM-dd HH:mm:ss SSS"]  
}  
mutate { remove\_field =\> "thread" }  
mutate { add\_field =\> { log\_type =\> "sql" } }  
}  
else{  
drop{}  
}  
}

While with this one , the logstash can just collect the data before 1 am  
filter {  
**if(" - request:" in [message]){**  
**grok {**  
**match =\> { "message" =\> "%{GREEDYDATA:timestamp} %{GREEDYDATA:thread} INFO - request:%{GREEDYDATA:serviceName} %{GREEDYDATA:serviceMethod} %{GREEDYDATA:sessionId} %{GREEDYDATA:ipAddress} %{GREEDYDATA:callerFrom} %{GREEDYDATA:operatorId} %{GREEDYDATA:operato rName} %{GREEDYDATA:customerId} %{GREEDYDATA:customerName} reqArgs:%{GREEDYDATA:reqArgs}"}**  
**}**  
**date {**  
**match =\> ["timestamp", "MM-dd HH:mm:ss SSS"]**  
**}**  
**mutate { remove\_field =\> "thread" }**  
**mutate { add\_field =\> { log\_type =\> "ksrequest" } }**  
**}else**  
if(" - response:" in [message]){  
grok {  
match =\> { "message" =\> "%{GREEDYDATA:timestamp} %{GREEDYDATA:thread} INFO - response:%{GREEDYDATA:serviceName} %{GREEDYDATA:serviceMethod} %{GREEDYDATA:sessionId} %{GREEDYDATA:ipAddress} %{GREEDYDATA:callerFrom} %{GREEDYDATA:operatorId} %{GREEDYDATA:operat orName} %{GREEDYDATA:customerId} %{GREEDYDATA:customerName} %{NUMBER:timeConsumed}ms"}  
}  
date {  
match =\> ["timestamp", "MM-dd HH:mm:ss SSS"]  
}  
mutate { remove\_field =\> "thread" }  
mutate { add\_field =\> { log\_type =\> "ksresponse" } }  
}else if(" - SQL Preparing: " in [message]){  
grok {  
match =\> { "message" =\> "%{GREEDYDATA:timestamp} %{GREEDYDATA:thread} DEBUG - SQL Preparing: %{GREEDYDATA:sql} args:%{GREEDYDATA:sqlArgs}"}  
}  
date {  
match =\> ["timestamp", "MM-dd HH:mm:ss SSS"]  
}  
mutate { remove\_field =\> "thread" }  
mutate { add\_field =\> { log\_type =\> "sql" } }  
}  
else{  
drop{}  
}  
}

 ![](https://us1.discourse-cdn.com/elastic/original/2X/8/8cb7dd4da9a3659b9253082bffadd4c63e1079de.png) ![](https://us1.discourse-cdn.com/elastic/original/2X/7/7bcb11f43e6f243b01178e8072945e4eb5e57bdb.png)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:35am UTC](https://discuss.elastic.co/t/can-not-parse-the-whole-file-but-stopped-at-1-am-logstash-2-4/62542/2 "2017-07-06T04:35:05Z")

</div>

Closed on July 6, 2017, 4:35am UTC