Need to use Multiline with Filebeat Configuration on Specific Tags

Elastic Stack Beats
1

Hi Team,

We are using filebeat and have set up multiple prospectors in its configuration. We are also using multiline in it, now we want to add one more prospector but do not want it to use multiline configuration.
Here is my config file-

#=========================== Filebeat prospectors=============================
filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level,so

# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.
- input_type: log

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
   # - /var/log/applogs/*.log
   - /tmp/servicelogs/*/applogs/Logs/*.log
  tags: ["webapp-log"]

- input_type: log
  paths:
    - /tmp/servicelogs/*/tomcatlogs/*.log
  tags: ["tomcat-log"]

- input_type: log
  paths:
    - /tmp/servicelogs/le-standalone.log
  tags: ["le-log"]

  multiline.pattern: '^\{'
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 500           

#================================ Outputs=====================================      
#----------------------------- Logstash output--------------------------------

output.logstash:
  # The Logstash hosts
  enabled: true
  hosts: ["logstash.dxp-dev-infra.svc:5044"]

I do not want the Logs with the tag "le-log" to have multiline configuration, while rest should have.
Please suggest is there someway to do this. Thanks in advance!

2

Hello @manya12, I took the liberty to add quote to format your yaml configuration.

In you current configuration I see 3 differents inputs, multiline is an input configuration so you will have to define it at the input level. So if I understand correctly the following should work for you:

- input_type: log
  paths:
   - /tmp/servicelogs/*/applogs/Logs/*.log
  tags: ["webapp-log"]
  multiline.pattern: '^\{'
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 500   

- input_type: log
  paths:
    - /tmp/servicelogs/*/tomcatlogs/*.log
  tags: ["tomcat-log"]
  multiline.pattern: '^\{'
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 500   

- input_type: log
  paths:
    - /tmp/servicelogs/le-standalone.log
  tags: ["le-log"]

Multiline will be only used for the first 2 inputs and the last one will not have any multiline.

3

Thanks @pierhugues. It works perfect for me.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.