# Logstash File pattern are more , how to reduce the number of patterns **URL:** \ **Category:** Elasticsearch\ **Created:** [February 11, 2019, 1:16pm UTC](https://discuss.elastic.co/t/logstash-file-pattern-are-more-how-to-reduce-the-number-of-patterns/167861 "2019-02-11T13:16:25Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![priya08](https://avatars.discourse-cdn.com/v4/letter/p/2bfe46/32.png) [@priya08](https://discuss.elastic.co/u/priya08)\ **Post date:** [February 11, 2019, 1:16pm UTC](https://discuss.elastic.co/t/logstash-file-pattern-are-more-how-to-reduce-the-number-of-patterns/167861/1 "2019-02-11T13:16:25Z")
Hi Team, We are using logstash file , in that we are having so many Grok patterns , the lines of file is very huge , for debugging we are facing issue. Are we having any other option ? ---
**Author:** ![shyamari](https://avatars.discourse-cdn.com/v4/letter/s/8e8cbc/32.png) [@shyamari](https://discuss.elastic.co/u/shyamari)\ **Post date:** [February 11, 2019, 1:27pm UTC](https://discuss.elastic.co/t/logstash-file-pattern-are-more-how-to-reduce-the-number-of-patterns/167861/2 "2019-02-11T13:27:07Z")
Hi Priya, You could use below link for the information. > [@Logstash plus filebeat pipeline](https://discuss.elastic.co/t/logstash-plus-filebeat-pipeline/162282): > > Hi there, I have a api log file with the following format ... 2018.27.12 17:37:28.423 GET /api/v1/catalogManagement/productOffering 200 4 2018.27.12 17:37:28.242 GET /api/v1/addressManagement/address 200 1214 ... I would like to extract all the fields from log file and parse with grok to setup the @timestamp according first two records in each row Thanks! ---
**Author:** ![priya08](https://avatars.discourse-cdn.com/v4/letter/p/2bfe46/32.png) [@priya08](https://discuss.elastic.co/u/priya08)\ **Post date:** [February 11, 2019, 1:51pm UTC](https://discuss.elastic.co/t/logstash-file-pattern-are-more-how-to-reduce-the-number-of-patterns/167861/3 "2019-02-11T13:51:43Z")
Hi shyamari, Sample logstash file patterns ``` grok{ match => {"message" => "\[%{WORD:logLevel}\] %{WORD:logType}->\|datetime:%{NOTSPACE:time}\+0000\|hostname:%{HOSTNAME:hostname}/%{IPV4:ip}\(%{IPV4}\)\|threadId:%{NOTSPACE:threadId}\|userId:%{WORD:userId}\|id:%{NOTSPACE:id}\|applicationName:%{NOTSPACE:applicationName}\|className:%{NOTSPACE:className}\|logMessage:SERVICE_DEPENDENCY: CALL-COMPLETED From %{NOTSPACE:fromService} To %{NOTSPACE:toService} on Uri %{URI:targetUri} with latency %{NUMBER:latency}"} add_tag => ["service_dependency"] } if "_grokparsefailure" in [tags]{ grok{ remove_tag => ["_grokparsefailure"] match => {"message" => "\[%{WORD:logLevel}\] %{WORD:logType}->\|datetime:%{NOTSPACE:time}\+0000\|hostname:%{HOSTNAME:hostname}/%{IPV4:ip}\(%{IPV4}\)\|threadId:%{NOTSPACE:threadId}\|userId:%{WORD:userId}\|id:%{NOTSPACE:id}\|applicationName:%{NOTSPACE:applicationName}\|className:%{NOTSPACE:className}\|logMessage:%{GREEDYDATA:logMessage}"} add_tag => ["applog"] } } if "_grokparsefailure" in [tags]{ grok{ remove_tag => ["_grokparsefailure"] match => {"message" => "\[%{WORD:logLevel}\] %{WORD:logType}->\|datetime:%{NOTSPACE:time}\+0000\|hostname:%{HOSTNAME:hostname}/%{IPV4:ip}\(%{IPV4}\)\|threadId:%{NOTSPACE:threadId}\|id:%{NOTSPACE:id}\|className:%{NOTSPACE:className}\|logMessage:%{GREEDYDATA:logMessage}"} add_tag => ["applog"] } } if "_grokparsefailure" in [tags]{ grok{ remove_tag => ["_grokparsefailure"] match => {"message" => "\[%{WORD:logLevel}\] %{WORD:logType}->\|datetime:%{NOTSPACE:time}\+0000\|hostname:%{HOSTNAME:hostname}/%{IPV4:ip}\(%{IPV4}\)\|threadId:%{NOTSPACE:threadId}\|userId:%{WORD:userId}\|id:%{NOTSPACE:id}\|applicationName:%{NOTSPACE:applicationName}\|className:%{NOTSPACE:className}"} add_tag => ["applog"] } } ``` like this i am having so many patterns , because every log message comes with different fields. ---
**Author:** ![shyamari](https://avatars.discourse-cdn.com/v4/letter/s/8e8cbc/32.png) [@shyamari](https://discuss.elastic.co/u/shyamari)\ **Post date:** [February 11, 2019, 2:18pm UTC](https://discuss.elastic.co/t/logstash-file-pattern-are-more-how-to-reduce-the-number-of-patterns/167861/4 "2019-02-11T14:18:55Z")
You could try below by passing different log messages: echo "2018-12-07 15:19:03" | logstash -e 'input { stdin {} } filter { date { match =\> ["message", "yyyy-MM-dd HH:mm:ss"] } }' or you could make file with logs you would like to test. cat file.log | logstash -e 'input { stdin {} } filter { date { match =\> ["message", "yyyy-MM-dd HH:mm:ss"] } }' ---
**Author:** ![priya08](https://avatars.discourse-cdn.com/v4/letter/p/2bfe46/32.png) [@priya08](https://discuss.elastic.co/u/priya08)\ **Post date:** [February 12, 2019, 10:00am UTC](https://discuss.elastic.co/t/logstash-file-pattern-are-more-how-to-reduce-the-number-of-patterns/167861/5 "2019-02-12T10:00:07Z")
Thanks for reply, But that was not my query , in my application i am having so many log statements , for each statement i have written one pattern. Their is any chance to create all the fields will come in log statements , so we can least bother about the pattern is existing in logstash or not? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [February 12, 2019, 10:55am UTC](https://discuss.elastic.co/t/logstash-file-pattern-are-more-how-to-reduce-the-number-of-patterns/167861/6 "2019-02-12T10:55:49Z")
Can you provide a sample of your message? For what I saw in your grok patterns, maybe your messages could be parsed using a KV filter, but you need to provide a sample of your message for better help. ---
**Author:** ![priya08](https://avatars.discourse-cdn.com/v4/letter/p/2bfe46/32.png) [@priya08](https://discuss.elastic.co/u/priya08)\ **Post date:** [February 13, 2019, 9:50am UTC](https://discuss.elastic.co/t/logstash-file-pattern-are-more-how-to-reduce-the-number-of-patterns/167861/7 "2019-02-13T09:50:45Z")
Thank you so much ---
**Author:** ![priya08](https://avatars.discourse-cdn.com/v4/letter/p/2bfe46/32.png) [@priya08](https://discuss.elastic.co/u/priya08)\ **Post date:** [February 14, 2019, 1:19pm UTC](https://discuss.elastic.co/t/logstash-file-pattern-are-more-how-to-reduce-the-number-of-patterns/167861/8 "2019-02-14T13:19:06Z")
Hi , in logstash i have created the KV filter , application level log statements are coming as expect but i am getting below error in logstash console . [2019-02-13T18:47:24,648][WARN][org.logstash.FieldReference] Detected ambiguous Field Reference `[I NFO] Tomcat started on port(s)`, which we expanded to the path `[INFO, Tomcat started on port(s)]`; in a future release of Logstash, ambiguous Field References will not be expanded. This log coming from spring boot level , how to control this error. ---
**Author:** ![scathatheworm](https://avatars.discourse-cdn.com/v4/letter/s/d78d45/32.png) [@scathatheworm](https://discuss.elastic.co/u/scathatheworm)\ **Post date:** [February 14, 2019, 1:30pm UTC](https://discuss.elastic.co/t/logstash-file-pattern-are-more-how-to-reduce-the-number-of-patterns/167861/9 "2019-02-14T13:30:20Z")
It seems to me you should be using an array of patterns to match stuff, although you loose the tagging you are doing, but you can work that out another way. ``` grok { match => { "message" => [ "Duration: %{NUMBER:duration}", "Speed: %{NUMBER:speed}" ] } } ``` See [https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#plugins-filters-grok-match](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#plugins-filters-grok-match) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 14, 2019, 1:30pm UTC](https://discuss.elastic.co/t/logstash-file-pattern-are-more-how-to-reduce-the-number-of-patterns/167861/10 "2019-03-14T13:30:22Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.