# Large amount of Grok patterns

**URL:** <https://discuss.elastic.co/t/large-amount-of-grok-patterns/60781>\
**Category:** Logstash\
**Created:** [September 17, 2016, 10:47am UTC](https://discuss.elastic.co/t/large-amount-of-grok-patterns/60781 "2016-09-17T10:47:30Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![jaykumar](https://avatars.discourse-cdn.com/v4/letter/j/439d5e/32.png) [@jaykumar](https://discuss.elastic.co/u/jaykumar)\
**Post date:** [September 17, 2016, 10:47am UTC](https://discuss.elastic.co/t/large-amount-of-grok-patterns/60781/1 "2016-09-17T10:47:30Z")

</div>

Hi All,

I am trying to figure out best way to do event classification.  
In my case, below is sample event:

Jul 21 01:19:57.58 172.20.20.100 date=2016-07-20 time=20:12:25 timezone="UTC" device\_name="X2123" device\_id=C032302-2323 log\_id=98393783 log\_type="Event" log\_component="GUI" log\_subtype="Admin" status="Successful" priority=Notice user\_name="admin" src\_ip=172.31.1.1 dmsg="Appliance Access Settings were changed by 'admin' from '10.0.0.26' using 'GUI'"

I would like to classify events based on content of dmsg. I have almost 1000 events each with unique content in dmsg. I have written 1000 grok pattern for each such unique message. Below is one of the example:

PID107 Appliance Access Settings were changed by %{QSSTRING} from %{QSSTRING} using %{QSSTRING}

Another example:

PID144 Service %{QSSTRING} was started by %{QSSTRING} from %{QSSTRING} using %{QSSTRING}

I have created configuration file like:

```
		if "unclassified" in [tags] and [dmsg] {
			grok { patterns_dir => ["/etc/logstash-indexer/patterns"]
				match => { "dmsg" => "%{PID107}" }
				add_field => { "eventid" => "PID107" }
				remove_tag => "unclassified"
			}
		}

		if "unclassified" in [tags] and [dmsg] {
			grok { patterns_dir => ["/etc/logstash-indexer/patterns"]
				match => { "dmsg" => "%{PID144}" }
				add_field => { "eventid" => "PID144" }
				remove_tag => "unclassified"
			}
		}

		if [eventid] {
		
			translate {
				field => "eventid"
				destination => "eventtype"
				dictionary_path => "/data/eventtype.csv"
				fallback => "unknown"
			}
                  }

```

Above works perfectly fine. I have added remaining 1000 grok pattern match filters like:

```
			grok { patterns_dir => ["/etc/logstash-indexer/patterns"]
				match => { "dmsg" => "%{PID1000}" }
				add_field => { "eventid" => "PID1000" }
				remove_tag => "unclassified"
			}

```

I am stuck here!

Logstash takes 5-6 minutes to load configuration and then it gives error:

/opt/logstash/bin/logstash -f /etc/logstash-indexer/conf.d/ -t  
java.lang.OutOfMemoryError: Java heap space  
Dumping heap to /opt/logstash/heapdump.hprof ...  
Unable to create /opt/logstash/heapdump.hprof: File exists  
Error: Your application used more memory than the safety cap of 1G.  
Specify -J-Xmx####m to increase it (#### = cap size in MB).  
Specify -w for full OutOfMemoryError stack trace

I can increase memory but is this the best way to classify 1000+ events? Any suggestions with better approach?

Thank you.

Jay

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 17, 2016, 5:57pm UTC](https://discuss.elastic.co/t/large-amount-of-grok-patterns/60781/2 "2016-09-17T17:57:02Z")

</div>

Just capture the first token ("PIDxxx") into the `eventid` field and you only need a single grok filter. In other words, change the beginning of each grok pattern to e.g. `%{WORD:eventid}`. But I don't get why you need to match the whole strings at all. Why not just grab the event id from the string and call it a day? You don't appear to be extract stuff from the message anyway so I don't see how it matterns if "PID107" is followed by "Appliance Access Settings ..." or something else. Logstash is a log _processor_ not a _validator_.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:38am UTC](https://discuss.elastic.co/t/large-amount-of-grok-patterns/60781/3 "2017-07-06T04:38:05Z")

</div>