# Parsing multiline log options

**URL:** <https://discuss.elastic.co/t/parsing-multiline-log-options/256441>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [November 24, 2020, 5:20am UTC](https://discuss.elastic.co/t/parsing-multiline-log-options/256441 "2020-11-24T05:20:17Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![newmember](https://avatars.discourse-cdn.com/v4/letter/n/ce7236/32.png) [@newmember](https://discuss.elastic.co/u/newmember)\
**Post date:** [November 24, 2020, 5:20am UTC](https://discuss.elastic.co/t/parsing-multiline-log-options/256441/1 "2020-11-24T05:20:17Z")

</div>

I would like to ingest this multiline event and capture some fields;

1. 

Is it best to capture the fields at the host?  
or with logstash as part of a filter?

1. 

how would I set up regex to catch TTC\_UID KVP and TCCRON KVP?

I am not sure how to search how to search past the end of the first line and down until the two pairs I am wanting to capture.  
pattern   
^d{2}:\d{2}:\d{2}.\d{3}  
negate TRUE   
match after

```auto
15:57:07.935 Int 03445 message "Saved" received from host_name ("12@domain12.url")
	 : message Event
	AttributeCallState	0
	TCCRCaType	7
	TCCRProType	4
	TCCRCaID	16782423737
	TCCRCoID	"qweeeee"
	TCCR_UID	'dfdf_tyyy0023432423'
	TCCRDIS	'zx90'
	TCCRNI	'998765'
	TCCRTDN	'12456'
	TCCRAID	'bsmyth'
	TCCRTole	2
	TCCRThue	'4456'
	TCCRPID	'150001234'
	TCCRON	'4012721'
	TCCRRole	1
	TCCREs	[45] 00 44 00 00..
		'OtherName'	'Dave'
		'BusinessCall'	1

```

---

<div class="post-metadata">

**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\
**Post date:** [November 25, 2020, 1:50am UTC](https://discuss.elastic.co/t/parsing-multiline-log-options/256441/2 "2020-11-25T01:50:08Z")

</div>

> [@newmember](#):
>
> Is it best to capture the fields at the host?  
> or with logstash as part of a filter?

It depends. If you are able to parse the message at the host itself (or even in Elasticsearch using an [Ingest Node pipeline](https://www.elastic.co/guide/en/beats/filebeat/master/configuring-ingest-node.html)), you can reduce the complexity in your architecture a bit by not needing another component — Logstash — in between Filebeat and Elasticsearch. However, there are some transformations and enrichments only possible in Logstash, at which point you might as well use it to do all the parsing and use Filebeat just as a fast shipper.

> [@newmember](#):
>
> how would I set up regex to catch TTC\_UID KVP and TCCRON KVP?

In combination with Filebeat multiline options, you probably want to look into using the [`dissect` processor](https://www.elastic.co/guide/en/beats/filebeat/current/dissect.html) in Filebeat or the [`grok` processor](https://www.elastic.co/guide/en/elasticsearch/reference/current/grok-processor.html) in an Ingest Node pipeline.

---

<div class="post-metadata">

**Author:** ![newmember](https://avatars.discourse-cdn.com/v4/letter/n/ce7236/32.png) [@newmember](https://discuss.elastic.co/u/newmember)\
**Post date:** [November 29, 2020, 8:34am UTC](https://discuss.elastic.co/t/parsing-multiline-log-options/256441/3 "2020-11-29T08:34:01Z")

</div>

Thank you.

I do have Logstash set up to take the load from the remote hosts. Remote hosts are across the internet and Logstash and ES are on AWS in different regions.

I few things I will be testing are the cpu load of pipelines on the remote hosts, I wouldnt want to drain the vms of all their cpu for log processing.  
If I want to do updates I am thinking its easier for me to push changes to logstash on aws instead of at the the remote hosts. So operationally it would be quicker if its centralized.

So if I have pass a few KVPs with the event a t a low cpu cost on the remote I think I can be okay.  
Those two KVPs dont change so operationally I think I can be safe for awhile.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 27, 2020, 10:34am UTC](https://discuss.elastic.co/t/parsing-multiline-log-options/256441/4 "2020-12-27T10:34:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.