# Grok for code to get formatted output **URL:** \ **Category:** Logstash\ **Created:** [May 4, 2016, 7:48am UTC](https://discuss.elastic.co/t/grok-for-code-to-get-formatted-output/49132 "2016-05-04T07:48:33Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![prakashkumar](https://avatars.discourse-cdn.com/v4/letter/p/b5ac83/32.png) [@prakashkumar](https://discuss.elastic.co/u/prakashkumar)\ **Post date:** [May 4, 2016, 7:48am UTC](https://discuss.elastic.co/t/grok-for-code-to-get-formatted-output/49132/1 "2016-05-04T07:48:33Z")
#My filebeat0.yml filebeat: ## prospectors: paths: - c:/Users/akashsoni/Downloads/k/logstash-tutorial/tutorial/_.log - c:/Users/akashsoni/Downloads/k/logstash-tutorial/tutorial/_.err input\_type: log registry\_file: "C:/ProgramData/filebeat/registry" output: logstash: ``` hosts: ["localhost:5044"] ``` shipper: logging: files: ``` rotateeverybytes: 10485760 # = 10MB ``` * * * #my logstash.conf input { beats { port =\> 5044 } } filter { grok { \*\*#what should i return here so i will get 'Exception','warning','Error',likes word in my log ,so \*\* \*\* #only those file who has 'Exception','warning','Error',likes word in log \*\* **# have to store it to my elasticsearch ,otherwise not** } } output { elasticsearch { hosts =\> "localhost:9200" manage\_template =\> false index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}" document\_type =\> "%{[@metadata][type]}" } } * * * My log files are of different format one is of java and other is of nginx store at single folder one of log file is having size of 19.2gb. #My Java log : javax.mail.internet.AddressException: Missing local name in string ``@gmail.com'' at javax.mail.internet.InternetAddress.checkAddress(InternetAddress.java:1209) ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 4, 2016, 8:43am UTC](https://discuss.elastic.co/t/grok-for-code-to-get-formatted-output/49132/2 "2016-05-04T08:43:15Z")
So you want to capture events with "exception", "warning", or "error" anywhere in the message and send them to Elasticsearch and drop all other events? ---
**Author:** ![prakashkumar](https://avatars.discourse-cdn.com/v4/letter/p/b5ac83/32.png) [@prakashkumar](https://discuss.elastic.co/u/prakashkumar)\ **Post date:** [May 4, 2016, 8:46am UTC](https://discuss.elastic.co/t/grok-for-code-to-get-formatted-output/49132/3 "2016-05-04T08:46:19Z")
yup, it is possible through logstash? ---
**Author:** ![prakashkumar](https://avatars.discourse-cdn.com/v4/letter/p/b5ac83/32.png) [@prakashkumar](https://discuss.elastic.co/u/prakashkumar)\ **Post date:** [May 4, 2016, 8:49am UTC](https://discuss.elastic.co/t/grok-for-code-to-get-formatted-output/49132/4 "2016-05-04T08:49:33Z")
yes tell me the code and where to apply those code to achieve required goal . ---
**Author:** ![prakashkumar](https://avatars.discourse-cdn.com/v4/letter/p/b5ac83/32.png) [@prakashkumar](https://discuss.elastic.co/u/prakashkumar)\ **Post date:** [May 4, 2016, 10:00am UTC](https://discuss.elastic.co/t/grok-for-code-to-get-formatted-output/49132/5 "2016-05-04T10:00:24Z")
sir, will you help me on that? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 4, 2016, 10:37am UTC](https://discuss.elastic.co/t/grok-for-code-to-get-formatted-output/49132/6 "2016-05-04T10:37:24Z")
Please have patience, we are all volunteers here and someone will respond to you when they can. ---
**Author:** ![prakashkumar](https://avatars.discourse-cdn.com/v4/letter/p/b5ac83/32.png) [@prakashkumar](https://discuss.elastic.co/u/prakashkumar)\ **Post date:** [May 4, 2016, 10:48am UTC](https://discuss.elastic.co/t/grok-for-code-to-get-formatted-output/49132/7 "2016-05-04T10:48:06Z")
okay sir, ---
**Author:** ![fbaligand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fbaligand/32/5657_2.png) [@fbaligand](https://discuss.elastic.co/u/fbaligand)\ **Post date:** [May 4, 2016, 5:47pm UTC](https://discuss.elastic.co/t/grok-for-code-to-get-formatted-output/49132/8 "2016-05-04T17:47:42Z")
The easy way I see to have only messages containing 'Exception','warning','Error' sent to elasticsearch : ``` output { if [message] =~ "Exception|warning|Error" { elasticsearch { hosts => "localhost:9200" manage_template => false index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" document_type => "%{[@metadata][type]}" } } } ``` So need to grok filter ---
**Author:** ![prakashkumar](https://avatars.discourse-cdn.com/v4/letter/p/b5ac83/32.png) [@prakashkumar](https://discuss.elastic.co/u/prakashkumar)\ **Post date:** [May 5, 2016, 5:30am UTC](https://discuss.elastic.co/t/grok-for-code-to-get-formatted-output/49132/9 "2016-05-05T05:30:43Z")
thankyou sir, But how to grok to use this output code? ---
**Author:** ![fbaligand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fbaligand/32/5657_2.png) [@fbaligand](https://discuss.elastic.co/u/fbaligand)\ **Post date:** [May 5, 2016, 7:26am UTC](https://discuss.elastic.co/t/grok-for-code-to-get-formatted-output/49132/10 "2016-05-05T07:26:18Z")
Here the configuration you can do : ``` filter { grok { match => { "message" => "(?Exception|warning|Error)" } } if ![message_type] { drop {} } } ``` ---
**Author:** ![prakashkumar](https://avatars.discourse-cdn.com/v4/letter/p/b5ac83/32.png) [@prakashkumar](https://discuss.elastic.co/u/prakashkumar)\ **Post date:** [May 5, 2016, 7:29am UTC](https://discuss.elastic.co/t/grok-for-code-to-get-formatted-output/49132/11 "2016-05-05T07:29:48Z")
thank you sir for giving yours valuable time 🙂 ---
**Author:** ![prakashkumar](https://avatars.discourse-cdn.com/v4/letter/p/b5ac83/32.png) [@prakashkumar](https://discuss.elastic.co/u/prakashkumar)\ **Post date:** [May 5, 2016, 7:55am UTC](https://discuss.elastic.co/t/grok-for-code-to-get-formatted-output/49132/12 "2016-05-05T07:55:09Z")
## PS C:\Program Files\Filebeat\> ./filebeat -e -c filebeat0.yml -d "publish" 2016/05/05 07:26:16.287762 geolite.go:24: INFO GeoIP disabled: No paths were set under output.geoip.paths 2016/05/05 07:26:16.287762 logstash.go:106: INFO Max Retries set to: 3 2016/05/05 07:26:16.292766 outputs.go:126: INFO Activated logstash as output plugin. 2016/05/05 07:26:16.292766 publish.go:232: DBG Create output worker 2016/05/05 07:26:16.292766 publish.go:274: DBG No output is defined to store the topology. The server fields might not be filled. 2016/05/05 07:26:16.292766 publish.go:288: INFO Publisher name: akash 2016/05/05 07:26:16.297770 async.go:78: INFO Flush Interval set to: 1s 2016/05/05 07:26:16.297770 async.go:84: INFO Max Bulk Size set to: 2048 2016/05/05 07:26:16.297770 async.go:92: DBG create bulk processing worker (interval=1s, bulk size=2048) 2016/05/05 07:26:16.298770 beat.go:147: INFO Init Beat: filebeat; Version: 1.2.1 2016/05/05 07:26:16.298770 beat.go:173: INFO filebeat sucessfully setup. Start running. 2016/05/05 07:26:16.299771 registrar.go:66: INFO Registry file set to: C:\Program Files\Filebeat.filebeat 2016/05/05 07:26:16.299771 registrar.go:76: INFO Loading registrar data from C:\Program Files\Filebeat.filebeat 2016/05/05 07:26:16.332015 prospector.go:132: INFO Set ignore\_older duration to 0 2016/05/05 07:26:16.332015 prospector.go:132: INFO Set close\_older duration to 1h0m0s 2016/05/05 07:26:16.333016 prospector.go:132: INFO Set scan\_frequency duration to 10s 2016/05/05 07:26:16.333016 prospector.go:89: INFO Invalid input type set: 2016/05/05 07:26:16.333016 prospector.go:92: INFO Input type set to: log 2016/05/05 07:26:16.333016 prospector.go:132: INFO Set backoff duration to 1s 2016/05/05 07:26:16.333016 prospector.go:132: INFO Set max\_backoff duration to 10s 2016/05/05 07:26:16.333016 prospector.go:112: INFO force\_close\_file is disabled 2016/05/05 07:26:16.333016 prospector.go:132: INFO Set ignore\_older duration to 0 2016/05/05 07:26:16.333016 prospector.go:132: INFO Set close\_older duration to 1h0m0s 2016/05/05 07:26:16.334016 prospector.go:132: INFO Set scan\_frequency duration to 10s 2016/05/05 07:26:16.334016 prospector.go:89: INFO Invalid input type set: err 2016/05/05 07:26:16.334016 prospector.go:92: INFO Input type set to: log 2016/05/05 07:26:16.334016 prospector.go:132: INFO Set backoff duration to 1s 2016/05/05 07:26:16.334016 prospector.go:132: INFO Set max\_backoff duration to 10s 2016/05/05 07:26:16.334016 prospector.go:112: INFO force\_close\_file is disabled 2016/05/05 07:26:16.334016 prospector.go:142: INFO Starting prospector of type: log 2016/05/05 07:26:16.333016 spooler.go:77: INFO Starting spooler: spool\_size: 2048; idle\_timeout: 5s 2016/05/05 07:26:16.335018 prospector.go:142: INFO Starting prospector of type: log 2016/05/05 07:26:16.356031 log.go:113: INFO Harvester started for file: c:\Users\akashsoni\Downloads\k\logstash-tutorial \tutorial\20160402000001.err 2016/05/05 07:26:16.357032 log.go:113: INFO Harvester started for file: c:\Users\akashsoni\Downloads\k\logstash-tutorial \tutorial\20160402000101.err 2016/05/05 07:26:16.358033 log.go:113: INFO Harvester started for file: c:\Users\akashsoni\Downloads\k\logstash-tutorial \tutorial\20160402000201.err 2016/05/05 07:26:16.359033 log.go:113: INFO Harvester started for file: c:\Users\akashsoni\Downloads\k\logstash-tutorial \tutorial\20160402000401.err 2016/05/05 07:26:16.359033 log.go:113: INFO Harvester started for file: c:\Users\akashsoni\Downloads\k\logstash-tutorial \tutorial\20160402000301.err 2016/05/05 07:26:16.360034 log.go:113: INFO Harvester started for file: c:\Users\akashsoni\Downloads\k\logstash-tutorial \tutorial\20160402000601.err 2016/05/05 07:26:16.360034 log.go:113: INFO Harvester started for file: c:\Users\akashsoni\Downloads\k\logstash-tutorial \tutorial\20160402000502.err 2016/05/05 07:26:19.661256 crawler.go:78: INFO All prospectors initialised with 3064 states to persist 2016/05/05 07:26:19.663256 registrar.go:83: INFO Starting Registrar 2016/05/05 07:26:19.664257 publish.go:88: INFO Start sending events to output This is my filebeat output : courser is blinking continuously but not showing output why ? when i use same code as i written above in filebeat config file with .log extension it will work fine. tomorrow this code is running for both err and log why? ---
**Author:** ![fbaligand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fbaligand/32/5657_2.png) [@fbaligand](https://discuss.elastic.co/u/fbaligand)\ **Post date:** [May 5, 2016, 8:14am UTC](https://discuss.elastic.co/t/grok-for-code-to-get-formatted-output/49132/13 "2016-05-05T08:14:01Z")
Hi, It is a completely different problem, so I invite you to create a new topic for that. ---
**Author:** ![prakashkumar](https://avatars.discourse-cdn.com/v4/letter/p/b5ac83/32.png) [@prakashkumar](https://discuss.elastic.co/u/prakashkumar)\ **Post date:** [May 5, 2016, 8:16am UTC](https://discuss.elastic.co/t/grok-for-code-to-get-formatted-output/49132/14 "2016-05-05T08:16:45Z")
okay sir, i tried for two days for that grok and you solve it 🙂 from where can learn grok . thankyou again sir. ---
**Author:** ![fbaligand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fbaligand/32/5657_2.png) [@fbaligand](https://discuss.elastic.co/u/fbaligand)\ **Post date:** [May 5, 2016, 8:30am UTC](https://discuss.elastic.co/t/grok-for-code-to-get-formatted-output/49132/15 "2016-05-05T08:30:28Z")
You're welcome. You can learn more about grok here : [https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:59am UTC](https://discuss.elastic.co/t/grok-for-code-to-get-formatted-output/49132/16 "2017-07-06T04:59:19Z")