# Split long log into small parts and apply grok in logstash filter

**URL:** <https://discuss.elastic.co/t/split-long-log-into-small-parts-and-apply-grok-in-logstash-filter/103363>\
**Category:** Logstash\
**Created:** [October 10, 2017, 12:23pm UTC](https://discuss.elastic.co/t/split-long-log-into-small-parts-and-apply-grok-in-logstash-filter/103363 "2017-10-10T12:23:13Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![tharu85](https://avatars.discourse-cdn.com/v4/letter/t/e95f7d/32.png) [@tharu85](https://discuss.elastic.co/u/tharu85)\
**Post date:** [October 10, 2017, 12:23pm UTC](https://discuss.elastic.co/t/split-long-log-into-small-parts-and-apply-grok-in-logstash-filter/103363/1 "2017-10-10T12:23:13Z")

</div>

I have a log which cannot directly extract using grok patterns under logstash filter. I have tested full grok pattern for the log in grok debugger and it works fine. But when I run the grok pattern under logstash filter gives grok failure error.

So I need to break full message into small parts and apply the grok individually.

I need to know, whether above mention requirement is possible under logstash ?

If yes, need a guide line to perform.

---

<div class="post-metadata">

**Author:** ![CDR](https://avatars.discourse-cdn.com/v4/letter/c/d9b06d/32.png) [@CDR](https://discuss.elastic.co/u/CDR)\
**Post date:** [October 10, 2017, 1:00pm UTC](https://discuss.elastic.co/t/split-long-log-into-small-parts-and-apply-grok-in-logstash-filter/103363/2 "2017-10-10T13:00:39Z")

</div>

Do you know why the log is being marked as grokparse failure? Maybe we could help you solve that issue.

If you are dead set on splitting up your log, how are you reading the log in? Through a file? Syslog? If you are reading a the log in using the file input that has a configurable option called [delimiter](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-delimiter) which allows you to set a new line delimiter.

Perhaps you could attach a sample log, the grok you constructed and your logstash.conf file, removing any sensitive information.

---

<div class="post-metadata">

**Author:** ![tharu85](https://avatars.discourse-cdn.com/v4/letter/t/e95f7d/32.png) [@tharu85](https://discuss.elastic.co/u/tharu85)\
**Post date:** [October 11, 2017, 4:05am UTC](https://discuss.elastic.co/t/split-long-log-into-small-parts-and-apply-grok-in-logstash-filter/103363/3 "2017-10-11T04:05:48Z")

</div>

@CDR

Below is the grok patterns put under logstash filter.

```
filter {

grok {
	match => ["message", "<%{POSINT:pri_id}>%{SYSLOGTIMESTAMP:log_timestamp} %{HOSTNAME:hostname} %{WORD:source}: \[%{DATA:num}\] %{GREEDYDATA:signature} \[Impact: %{DATA:impact}\] From \\"%{DATA:device}\\" %{WORD:seq} %{WORD:day} %{SYSLOGTIMESTAMP:trigger_timestamp} %{DATA:list_year} %{WORD:time_zone} \[Classification: %{GREEDYDATA:classification}\] \[Priority: %{NUMBER:priority}\] \{%{DATA:protocol}\} (?<srcip>[0-9]+.[0-9]+.[0-9]+.[0-9]+|N/A):(?<srcport>[0-9]+|N/A) \(%{DATA:srcname}\)->(?<dstip>[0-9]+.[0-9]+.[0-9]+.[0-9]+|N/A):(?<dstport>[0-9]+|N/A) \(%{DATA:dstname}\)"]
  }	
}

```

and here are the sample log. Some sensitive information are masked here.

\<46\>Oct 10 11:44:29 XXX-XX-XX XXXXS: [135:2:2] internal: session established [Impact: Potentially Vulnerable] From "XXX-XX-XX" at Tue Oct 10 11:44:29 2017 UTC [Classification: A TCP Connection was Detected] [Priority: 3] {tcp} 192.168.1.1:23 (unknown)-\>10.10.10.10:XXXXX (unknown)

\<46\>Oct 10 11:44:32 XXX-XX-XX XXXXS: [119:4:1] http\_inspect: BARE BYTE UNICODE ENCODING [Impact: Potentially Vulnerable] From "XXX-XX-XX" at Tue Oct 10 11:44:31 2017 UTC [Classification: Not Suspicious Traffic] [Priority: 3] {tcp} [10.10.10.XXX](http://10.10.10.XXX):XXXXX (unknown)-\>192.168.1.1:80 (unknown)"

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 8, 2017, 4:05am UTC](https://discuss.elastic.co/t/split-long-log-into-small-parts-and-apply-grok-in-logstash-filter/103363/4 "2017-11-08T04:05:53Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.