Parsing multiline log options

Elastic Stack Beats
1

I would like to ingest this multiline event and capture some fields;

Is it best to capture the fields at the host?
or with logstash as part of a filter?

how would I set up regex to catch TTC_UID KVP and TCCRON KVP?

I am not sure how to search how to search past the end of the first line and down until the two pairs I am wanting to capture.
pattern
^d{2}:\d{2}:\d{2}.\d{3}
negate TRUE
match after

15:57:07.935 Int 03445 message "Saved" received from host_name ("12@domain12.url")
	 : message Event
	AttributeCallState	0
	TCCRCaType	7
	TCCRProType	4
	TCCRCaID	16782423737
	TCCRCoID	"qweeeee"
	TCCR_UID	'dfdf_tyyy0023432423'
	TCCRDIS	'zx90'
	TCCRNI	'998765'
	TCCRTDN	'12456'
	TCCRAID	'bsmyth'
	TCCRTole	2
	TCCRThue	'4456'
	TCCRPID	'150001234'
	TCCRON	'4012721'
	TCCRRole	1
	TCCREs	[45] 00 44 00 00..
		'OtherName'	'Dave'
		'BusinessCall'	1
2

It depends. If you are able to parse the message at the host itself (or even in Elasticsearch using an Ingest Node pipeline), you can reduce the complexity in your architecture a bit by not needing another component — Logstash — in between Filebeat and Elasticsearch. However, there are some transformations and enrichments only possible in Logstash, at which point you might as well use it to do all the parsing and use Filebeat just as a fast shipper.

In combination with Filebeat multiline options, you probably want to look into using the dissect processor in Filebeat or the grok processor in an Ingest Node pipeline.

3

Thank you.

I do have Logstash set up to take the load from the remote hosts. Remote hosts are across the internet and Logstash and ES are on AWS in different regions.

I few things I will be testing are the cpu load of pipelines on the remote hosts, I wouldnt want to drain the vms of all their cpu for log processing.
If I want to do updates I am thinking its easier for me to push changes to logstash on aws instead of at the the remote hosts. So operationally it would be quicker if its centralized.

So if I have pass a few KVPs with the event a t a low cpu cost on the remote I think I can be okay.
Those two KVPs dont change so operationally I think I can be safe for awhile.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.