# Grok filter for selecting and formatting certain logs lines **URL:** \ **Category:** Logstash\ **Created:** [May 15, 2018, 9:12am UTC](https://discuss.elastic.co/t/grok-filter-for-selecting-and-formatting-certain-logs-lines/131892 "2018-05-15T09:12:03Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Parthgandhi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/parthgandhi/32/22742_2.png) [@Parthgandhi](https://discuss.elastic.co/u/Parthgandhi)\ **Post date:** [May 15, 2018, 9:12am UTC](https://discuss.elastic.co/t/grok-filter-for-selecting-and-formatting-certain-logs-lines/131892/1 "2018-05-15T09:12:03Z")
HI Team, I am writing up a grok filter for parsing my application log which is unstructured. What i need is to look for certain lines and generate output in a specific format. e.g below are my logs > 2018-05-07 01:19:40 M :Memory (xivr = 513.2 Mb, system = 3502.0 Mb, physical = 5386.7 Mb), CpuLoad (sys = 0%, xivr = 0%) > 2018-05-07 01:29:40 M :Memory (xivr = 513.2 Mb, system = 3495.3 Mb, physical = 5370.1 Mb), CpuLoad (sys = 0%, xivr = 0%) > 2018-05-07 05:51:19 1 :Hangup call > **2018-05-07 05:51:22 24 :Answer call from 71840746 for 91783028 [C:\xivr\es\IVR-Dialin.dtx]** > 2018-05-07 05:51:30 24 :Hangup call > **2018-05-07 05:51:34 24 :Answer call from 71840746 for 91783028 [C:\xivr\es\IVR-Dialin.dtx]** > 2018-05-07 00:31:21 45 :Device Dialogic Digital dxxxB12C1 [gc60.dev - Dialogic (SDK 6.0) ver 3.0.702:11646] (ThreadID: 1FF0, DriverChannel: 44) > 2018-05-07 00:31:22 40 :Device Dialogic Digital dxxxB10C4 [gc60.dev - Dialogic (SDK 6.0) ver 3.0.702:11646] (ThreadID: 1B2C, DriverChannel: 39) I need to enter only highlighted lines in below format in my Kibana: Other lines should be simply ignored > ``` > Logtimestamp: 2018-05-07 05:51:22 > Channel_id: 24 > Source_number: 71840746 > Destination_Number: 91783028 > > ``` How can this be achieved? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 15, 2018, 9:16am UTC](https://discuss.elastic.co/t/grok-filter-for-selecting-and-formatting-certain-logs-lines/131892/2 "2018-05-15T09:16:42Z")
Use a grok filter that only matches and extracts fields from this exact kind of message. If a line doesn't match it'll get tagged แบith `_grokparsefailure`. You can then look for that tag in a conditional that wraps a drop filter (see [How to exclude bad output (lines not matching 'grok' pattern) from logstash?](https://discuss.elastic.co/t/how-to-exclude-bad-output-lines-not-matching-grok-pattern-from-logstash/40459) for an example). ---
**Author:** ![Parthgandhi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/parthgandhi/32/22742_2.png) [@Parthgandhi](https://discuss.elastic.co/u/Parthgandhi)\ **Post date:** [May 15, 2018, 9:50am UTC](https://discuss.elastic.co/t/grok-filter-for-selecting-and-formatting-certain-logs-lines/131892/4 "2018-05-15T09:50:19Z")
HI @magnusbaeck, Below is the link to the conf file i created based on your suggestion. Could you please help me verify if thats ok. or need any changes. [https://pastebin.com/xifXz0gf](https://pastebin.com/xifXz0gf) PS: I am using same pipeline to filter IIS and IVR logs. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 15, 2018, 9:54am UTC](https://discuss.elastic.co/t/grok-filter-for-selecting-and-formatting-certain-logs-lines/131892/5 "2018-05-15T09:54:05Z")
It looks reasonably correct but you'll have to verify it yourself anyway. ---
**Author:** ![Parthgandhi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/parthgandhi/32/22742_2.png) [@Parthgandhi](https://discuss.elastic.co/u/Parthgandhi)\ **Post date:** [May 15, 2018, 10:43am UTC](https://discuss.elastic.co/t/grok-filter-for-selecting-and-formatting-certain-logs-lines/131892/6 "2018-05-15T10:43:20Z")
Sure. Crating a separate pipeline for IVR logs. ---
**Author:** ![Parthgandhi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/parthgandhi/32/22742_2.png) [@Parthgandhi](https://discuss.elastic.co/u/Parthgandhi)\ **Post date:** [May 15, 2018, 12:11pm UTC](https://discuss.elastic.co/t/grok-filter-for-selecting-and-formatting-certain-logs-lines/131892/7 "2018-05-15T12:11:40Z")
I am getting below error in filebeat. Is it has some thing to do with the logstash pipeline. | 2018-05-15T07:55:57.521-0400 | ERROR | logstash/async.go:235 | Failed to publish events caused by: write tcp 10.5.52.168:54592-\>x.x.x.x:5044: wsasend: An existing connection was forcibly closed by the remote host. | | --- | --- | --- | --- | | 2018-05-15T07:55:58.522-0400 | ERROR | pipeline/output.go:92 | Failed to publish events: write tcp 10.5.52.168:54592-\>x.x.x.x:5044: wsasend: An existing connection was forcibly closed by the remote host. | ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 15, 2018, 1:29pm UTC](https://discuss.elastic.co/t/grok-filter-for-selecting-and-formatting-certain-logs-lines/131892/8 "2018-05-15T13:29:51Z")
Have you configured Filebeat to use SSL but not Logstash? Or vice versa? ---
**Author:** ![Parthgandhi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/parthgandhi/32/22742_2.png) [@Parthgandhi](https://discuss.elastic.co/u/Parthgandhi)\ **Post date:** [May 17, 2018, 7:46am UTC](https://discuss.elastic.co/t/grok-filter-for-selecting-and-formatting-certain-logs-lines/131892/9 "2018-05-17T07:46:18Z")
vice versa. Below is my filebeat config for output. ``` output.logstash: # The Logstash hosts hosts: ["serverip:5044"] ``` ---
**Author:** ![Parthgandhi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/parthgandhi/32/22742_2.png) [@Parthgandhi](https://discuss.elastic.co/u/Parthgandhi)\ **Post date:** [May 28, 2018, 9:46am UTC](https://discuss.elastic.co/t/grok-filter-for-selecting-and-formatting-certain-logs-lines/131892/10 "2018-05-28T09:46:01Z")
@magnusbaeck are there any changes that i need to do? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 25, 2018, 9:46am UTC](https://discuss.elastic.co/t/grok-filter-for-selecting-and-formatting-certain-logs-lines/131892/11 "2018-06-25T09:46:04Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.