Grok filter for selecting and formatting certain logs lines

Elastic Stack Logstash
1

HI Team,
I am writing up a grok filter for parsing my application log which is unstructured. What i need is to look for certain lines and generate output in a specific format. e.g below are my logs

2018-05-07 01:19:40 M :Memory (xivr = 513.2 Mb, system = 3502.0 Mb, physical = 5386.7 Mb), CpuLoad (sys = 0%, xivr = 0%)
2018-05-07 01:29:40 M :Memory (xivr = 513.2 Mb, system = 3495.3 Mb, physical = 5370.1 Mb), CpuLoad (sys = 0%, xivr = 0%)
2018-05-07 05:51:19 1 :Hangup call
2018-05-07 05:51:22 24 :Answer call from 71840746 for 91783028 [C:\xivr\es\IVR-Dialin.dtx]
2018-05-07 05:51:30 24 :Hangup call
2018-05-07 05:51:34 24 :Answer call from 71840746 for 91783028 [C:\xivr\es\IVR-Dialin.dtx]
2018-05-07 00:31:21 45 :Device Dialogic Digital dxxxB12C1 [gc60.dev - Dialogic (SDK 6.0) ver 3.0.702:11646] (ThreadID: 1FF0, DriverChannel: 44)
2018-05-07 00:31:22 40 :Device Dialogic Digital dxxxB10C4 [gc60.dev - Dialogic (SDK 6.0) ver 3.0.702:11646] (ThreadID: 1B2C, DriverChannel: 39)

I need to enter only highlighted lines in below format in my Kibana: Other lines should be simply ignored

Logtimestamp:  2018-05-07 05:51:22    
Channel_id:  24 
Source_number: 71840746 
Destination_Number: 91783028

How can this be achieved?

2

Use a grok filter that only matches and extracts fields from this exact kind of message. If a line doesn't match it'll get tagged ẁith _grokparsefailure. You can then look for that tag in a conditional that wraps a drop filter (see How to exclude bad output (lines not matching 'grok' pattern) from logstash? for an example).

4

HI @magnusbaeck,

Below is the link to the conf file i created based on your suggestion. Could you please help me verify if thats ok. or need any changes.

https://pastebin.com/xifXz0gf

PS: I am using same pipeline to filter IIS and IVR logs.

5

It looks reasonably correct but you'll have to verify it yourself anyway.

6

Sure. Crating a separate pipeline for IVR logs.

7

I am getting below error in filebeat. Is it has some thing to do with the logstash pipeline.

2018-05-15T07:55:57.521-0400 ERROR logstash/async.go:235 Failed to publish events caused by: write tcp 10.5.52.168:54592->x.x.x.x:5044: wsasend: An existing connection was forcibly closed by the remote host.
2018-05-15T07:55:58.522-0400 ERROR pipeline/output.go:92 Failed to publish events: write tcp 10.5.52.168:54592->x.x.x.x:5044: wsasend: An existing connection was forcibly closed by the remote host.
8

Have you configured Filebeat to use SSL but not Logstash? Or vice versa?

9

vice versa. Below is my filebeat config for output.

output.logstash:
  # The Logstash hosts
  hosts: ["serverip:5044"]
10

@magnusbaeck are there any changes that i need to do?