Grok filter fails when met with new line in multiline log

[TheNmaptomyHeartBeat (Mohamed Salih)]

Hi,

I'm trying to parse some logs into Ealsticsearch from Filebeat.

The logs has a new line in them and their format is as follows:

# error 123
failed attempt because blah blah

I am changing the filter in the ingest pipeline for the system module. I have done this before for nginx module for some costume nginx logs and it works fine.

I tried a number of filters. These work on both the online grok builder and the one in DevTools in Kibana.

# %{GREEDYDATA:response}\n%{GREEDYDATA:error}

%{GREEDYDATA:response}\n%{GREEDYDATA:error}

However, the filter will always fail when ran through Filebeat. Furthermore, it also treats the logs are two separate entries when it's sent to Kibana. The first # error 123 will be a record by it self and then failed attempt because blah blah will be a record by itself and then obviously the filter will fail for both of them.

I can't use Logstash to send the data as I already have multiple logs from modules and custom logs going from Filebeat to Elasticsearch.

I tried adding the multiline options to system.yml but without any success.

  multiline.pattern: '^#'
  multiline.negate: true
  multiline.match: after

Should I add this to system.yml or filebeat.yml? and is it even right?

[Kaiyan_Sheng]

It should be in filebeat.yml: https://github.com/elastic/beats/blob/master/filebeat/filebeat.yml#L49

[TheNmaptomyHeartBeat (Mohamed Salih)]

Hi Kaiyan, thank you for your help.
I tried adding it to filebeat.yml but without any success.
Here is my filebeat.yml

####################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

# For more available modules and options, please see the filebeat.reference.yml sample
# configuration file.

#============================= Filebeat modules ===============================
filebeat.inputs:
#- type: log
#  enable: true
#  paths:
#    - /etc/test/error.log
multiline.pattern: "^# "
multiline.negate: true
multiline.match: after

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: /etc/filebeat/modules.d/*.yml

  # Set to true to enable config reloading
  #reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s


filebeat.overwrite_pipelines: true
#==================== Elasticsearch template setting ==========================
#
#setup.template.name: "filebeat-prod"
#setup.template.pattern: "filebeat-*"
#setup.template.settings:
#  index.number_of_shards: 0
#  index.number_of_replicas: 0

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["${HOST}"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  username: "${USR}"
  password: "${PWD}"
#  index: "filebeat-prod-%{[beat.version]}-%{+yyyy.MM.dd}"

#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: critical, error, warning, info, debug
#logging.level: info

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]

UPDATE:
it works when I specify the path on filebeat.yml. it sets the multiline flag and negates it too.
I had to enable paths and I had my spaces wrong.
But when I set the path on the system.yml file it still picks the logs as two lines and says that the grok filter has failed.

Is there a way to add a grok filter to the filebeat.inputs module? I can find system.yml /usr/share/filebeat but i had no luck doing the same for filebeat.input.