Merge lines

Elastic Stack Beats
1

Hi, i am new in elastic and i am working to send logs from filebeat to logstash, but i have a issue and i not found solution. the log have multiple related lines ID, information and status, i would like join them in single line to process more easy in elastic.

Please can you help me with my questions:

Is possible to do that i want?
what will be the procedure?

Regards

2

Are these lines 'intermixed' or always consecutive?
Any kind of start/stop pattern?

3

Hi, steffens, these lines are intermixed but always have the same pattern, for example

ID:
Address:
Enconding:
Content-Type:
Status:

i could exclude the lines that not are relevant for me.
exclude_lines: ['^Address','^Encoding','^Content-Type']

i would like have a single line with ID + status
Regards

4

Due to the lines being intermixed, but having an ID this is no plain multiline problem, but a correlation problem. You can try to solve this via aggregate filter in Logstash.

5

Hi, but how i can related the lines with the specific id, i have other line that say me if the record is inbound or outbound, a example is this

ID:
Address:
Enconding:
Content-Type:
Status:
Inbound:

ID:
Address:
Enconding:
Content-Type:
Status:
Outbound:

Regards

6

Assuming 3 events and lines being intermixed your log might look like this at worst:

id_1
address_1
id_2
address_2
encoding_2
encoding_1
id_3
content-type_1
address_3
encoding_3
content-type_3
status_1
content-type_2
inbound_1
status_2
inbound_2
status_3
inbound_3

The problem with this is: you can't even rely on order, as event_3 might surpass same lines from event_1 or event_2. In order to untangle these you need a correlation ID.

Your example suggest events are not intermixed, but written atomically. Like always:

id_1:
address_1:
encoding_1:
content-type_1:
status_1:
inbound_1:
id_2:
address_2:
encoding_2:
content-type_2:
status_2:
inbound_2:
id_3:
address_3:
encoding_3:
content-type_3:
status_3:
inbound_3:

If events are indeed written atomically, you can actually use multiline. Create a pattern looking for ^ID: (assuming ID is always the first entry) and use negate: true, match: after. See the table in the multiline docs.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.