# Grok filter fails when met with new line in multiline log **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [September 24, 2019, 10:29am UTC](https://discuss.elastic.co/t/grok-filter-fails-when-met-with-new-line-in-multiline-log/200837 "2019-09-24T10:29:45Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![TheNmaptomyHeartBeat](https://avatars.discourse-cdn.com/v4/letter/t/47e85d/32.png) [@TheNmaptomyHeartBeat](https://discuss.elastic.co/u/TheNmaptomyHeartBeat)\ **Post date:** [September 24, 2019, 10:29am UTC](https://discuss.elastic.co/t/grok-filter-fails-when-met-with-new-line-in-multiline-log/200837/1 "2019-09-24T10:29:45Z")
Hi, I'm trying to parse some logs into Ealsticsearch from Filebeat. The logs has a new line in them and their format is as follows: ```auto # error 123 failed attempt because blah blah ``` I am changing the filter in the ingest pipeline for the system module. I have done this before for nginx module for some costume nginx logs and it works fine. I tried a number of filters. These work on both the online grok builder and the one in DevTools in Kibana. ```auto # %{GREEDYDATA:response}\n%{GREEDYDATA:error} %{GREEDYDATA:response}\n%{GREEDYDATA:error} ``` However, the filter will always fail when ran through Filebeat. Furthermore, it also treats the logs are two separate entries when it's sent to Kibana. The first `# error 123` will be a record by it self and then `failed attempt because blah blah` will be a record by itself and then obviously the filter will fail for both of them. I can't use Logstash to send the data as I already have multiple logs from modules and custom logs going from Filebeat to Elasticsearch. I tried adding the multiline options to system.yml but without any success. ```auto multiline.pattern: '^#' multiline.negate: true multiline.match: after ``` Should I add this to `system.yml` or `filebeat.yml`? and is it even right? ---
**Author:** ![Kaiyan\_Sheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kaiyan_sheng/32/38247_2.png) [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)\ **Post date:** [September 24, 2019, 10:00pm UTC](https://discuss.elastic.co/t/grok-filter-fails-when-met-with-new-line-in-multiline-log/200837/2 "2019-09-24T22:00:17Z")
It should be in filebeat.yml: [https://github.com/elastic/beats/blob/master/filebeat/filebeat.yml#L49](https://github.com/elastic/beats/blob/master/filebeat/filebeat.yml#L49) ---
**Author:** ![TheNmaptomyHeartBeat](https://avatars.discourse-cdn.com/v4/letter/t/47e85d/32.png) [@TheNmaptomyHeartBeat](https://discuss.elastic.co/u/TheNmaptomyHeartBeat)\ **Post date:** [September 25, 2019, 12:28pm UTC](https://discuss.elastic.co/t/grok-filter-fails-when-met-with-new-line-in-multiline-log/200837/3 "2019-09-25T12:28:54Z")
Hi Kaiyan, thank you for your help. I tried adding it to filebeat.yml but without any success. Here is my filebeat.yml ```auto ####################### Filebeat Configuration Example ######################### # This file is an example configuration file highlighting only the most common # options. The filebeat.reference.yml file from the same directory contains all the # supported options with more comments. You can use it as a reference. # # You can find the full configuration reference here: # https://www.elastic.co/guide/en/beats/filebeat/index.html # For more available modules and options, please see the filebeat.reference.yml sample # configuration file. #============================= Filebeat modules =============================== filebeat.inputs: #- type: log # enable: true # paths: # - /etc/test/error.log multiline.pattern: "^# " multiline.negate: true multiline.match: after filebeat.config.modules: # Glob pattern for configuration loading path: /etc/filebeat/modules.d/*.yml # Set to true to enable config reloading #reload.enabled: false # Period on which files under path should be checked for changes #reload.period: 10s filebeat.overwrite_pipelines: true #==================== Elasticsearch template setting ========================== # #setup.template.name: "filebeat-prod" #setup.template.pattern: "filebeat-*" #setup.template.settings: # index.number_of_shards: 0 # index.number_of_replicas: 0 #================================ Outputs ===================================== # Configure what output to use when sending the data collected by the beat. #-------------------------- Elasticsearch output ------------------------------ output.elasticsearch: # Array of hosts to connect to. hosts: ["${HOST}"] # Optional protocol and basic auth credentials. #protocol: "https" username: "${USR}" password: "${PWD}" # index: "filebeat-prod-%{[beat.version]}-%{+yyyy.MM.dd}" #================================ Logging ===================================== # Sets log level. The default log level is info. # Available log levels are: critical, error, warning, info, debug #logging.level: info # At debug level, you can selectively enable logging only for some components. # To enable all selectors use ["*"]. Examples of other selectors are "beat", # "publish", "service". #logging.selectors: ["*"] ``` UPDATE: it works when I specify the path on filebeat.yml. it sets the multiline flag and negates it too. I had to enable paths and I had my spaces wrong. But when I set the path on the system.yml file it still picks the logs as two lines and says that the grok filter has failed. Is there a way to add a grok filter to the filebeat.inputs module? I can find system.yml `/usr/share/filebeat` but i had no luck doing the same for filebeat.input. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 23, 2019, 12:29pm UTC](https://discuss.elastic.co/t/grok-filter-fails-when-met-with-new-line-in-multiline-log/200837/4 "2019-10-23T12:29:23Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.