Use beats to read multiline event excluding some lines in-between

[lukiovas (Lukas Tučkus)]

I have a log:

 server.name 2021-03-28 10:03:28.648 INFO   ...
---------------------------
ID: 3974
Address: https://www...
Encoding: UTF-8
Http-Method: POST
Content-Type: text/xml
Headers: {Accept=[*/*], ...
Payload: <soap:Envelope...

I want filebeat to read this log as multiline event as so:

server.name 2021-03-28 10:03:28.648 INFO   ...
Payload: <soap:Envelope...

---, ID, Address, Encoding, Http-Method, Content-Type, Headers should be excluded. I tried following config in filebeat.yml unsuccessfully :

- type: log
  enabled: true

  paths:
    - "/var/log/app_servers/liferay-ext.log"
    - "/var/log/app_servers/liferay-int.log"

  fields :

    log_type: "logserver_prod_vrklt_liferay"
  
  exclude_lines: ['---','ID:','Address:','Encoding:','Http-Method:','Content-Type:','Headers:']

  multiline.pattern: ^\server.name
  multiline.match: after
  multiline.flush.pattern: 'Payload: '

Any ideas on how to get my desired result?

[jsoriano (Jaime Soriano)]

Hi @lukiovas, welcome to discuss

exclude_lines is executed after multiline groups the lines in a single field, so with the configuration you are trying I guess that lines are being grouped, but then they are excluded because they contain the pattens in exclude_lines.

I would suggest to use multiline to group all these lines, and then use processors to extract the relevant information. For example you could try to use dissect to extract the relevant fields, and then drop_fields to drop the original log message if you don't want to keep it.

[lukiovas (Lukas Tučkus)]

@jsoriano , thank you for your advice.

Firslty I'm trying to drop all the events that are not multiline. The strict yml configuration syntax is giving me a hard time. Could you give me an idea of whats wrong with the following processor:

processors:
   - drop_event:
       when:
          not:
          equals:
             flags: "multiline"

[jsoriano (Jaime Soriano)]

Be careful with the indentation, equals should have one more indentation level than not. Also, flags seems to be an array, and I am not sure if this condition works with arrays, you can also try with contains.

So try this:

processors:
   - drop_event:
       when:
         not:
           equals:
             flags: "multiline"

Or, as they are all objects with single members, you can simplify the condition like this:

processors:
   - drop_event:
       when.not.equals.flags: "multiline"

Try with contains instead of equals if it doesn't work after fixing indentation.