Filebeat configuration for multi line option (regex required)

amr.abdelfattah · July 10, 2019, 2:36pm

Kindly i need your urgent advise to configure filebeat to process below log:
2019-07-02 16:00:00.037 [SUBSCRIBER_PROFILE-1157917705-d73442b7-8d07-4aee-a850-09aa51ff37e2] Inquiry Item [com.etisalat.oms.inquiry.actions.Voi
ceMinutesCrossNetInquiry] Executed successfully******
2019-07-02 16:00:00.037 [SUBSCRIBER_PROFILE-1157917705-d73442b7-8d07-4aee-a850-09aa51ff37e2] Service Class [com.etisalat.oms.inquiry.actions.GeneralUnitsInquiry] l
oaded Successfully.
2019-07-02 16:00:00.037 [SUBSCRIBER_PROFILE-1157917705-d73442b7-8d07-4aee-a850-09aa51ff37e2] *************************** Starting Inquiry Item [com.etisalat.oms.in
quiry.actions.GeneralUnitsInquiry] *****************

i want each message in graylog start with date/time
for ex. below should be one event :

2019-07-02 16:00:00.400 [ELIGIBLE_PRODUCT-1122220199-4b666699-b9b0-4549-8aff-537b199e040d] Error While Processing Request [<?xml version="1.0" encoding="UTF-8"?>
    <eligibleProductsRequest>
        <channel>myEtisalatApp</channel>
        <extraRequestParameters>
            <name>INTERACTION_POINT</name>
            <value>meaHomePage</value>
        </extraRequestParameters>
        <extraRequestParameters>
            <name>APP_LANGUAGE</name>
            <value>EN</value>
        </extraRequestParameters>
        <extraRequestParameters>
            <name>APP_VERSION</name>
            <value>9</value>
        </extraRequestParameters>
        <requestType>BEST_OFFER</requestType>
        <transactionId>860acdaa2fc04ca88c7016d4f56b083d</transactionId>
        <msisdn>1122220199</msisdn>
    </eligibleProductsRequest>
    ].
     com.etisalat.oms.exception.ApplicationException: No eligible product  configured for this dial [1122220199]
            at com.etisalat.oms.inquiry.manager.GenericProductEligibility.postValidate(GenericProductEligibility.java:72) ~[oms.jar:?]
            at com.etisalat.oms.inquiry.manager.GenericProductEligibility.getEligibleProduct(GenericProductEligibility.java:42) ~[oms.jar:?]
            at com.etisalat.oms.interfaces.service.ProductManagementService.getEligibleProducts(ProductManagementService.java:179) ~[oms.jar:?]
            at com.etisalat.oms.interfaces.product.EligibleProductsHandler.handle(EligibleProductsHandler.java:47) [oms.jar:?]
            at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1115) [jetty-all-9.3.1.v20150714-uber.jar:9.3.1.v20150714]
            at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1051) [jetty-all-9.3.1.v20150714-uber.jar:9.3.1.v20150714]
            at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-all-9.3.1.v20150714-uber.jar:9.3.1.v20150714]
            at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) [jetty-all-9.3.1.v20150714-uber.jar:9.3.1.v20150714]
            at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119) [jetty-all-9.3.1.v20150714-uber.jar:9.3.1.v20150714]
            at org.eclipse.jetty.server.Server.handle(Server.java:517) [jetty-all-9.3.1.v20150714-uber.jar:9.3.1.v20150714]
            at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:302) [jetty-all-9.3.1.v20150714-uber.jar:9.3.1.v20150714]
            at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:242) [jetty-all-9.3.1.v20150714-uber.jar:9.3.1.v20150714]
            at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:238) [jetty-all-9.3.1.v20150714-uber.jar:9.3.1.v20150714]
            at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) [jetty-all-9.3.1.v20150714-uber.jar:9.3.1.v20150714]
            at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:57) [jetty-all-9.3.1.v20150714-uber.jar:9.3.1.v20150714]
            at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:213) [jetty-all-9.3.1.v20150714-uber.jar:9.3.1.v20150714]
            at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147) [jetty-all-9.3.1.v20150714-uber.jar:9.3.1.v20150714]
            at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) [jetty-all-9.3.1.v20150714-uber.jar:9.3.1.v20150714]
            at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) [jetty-all-9.3.1.v20150714-uber.jar:9.3.1.v20150714]
            at java.lang.Thread.run(Thread.java:745) [?:1.8.0_65]

I used below configuration ,but unfortunately ,didnt work,as per attached graylog screenshots

:

   multiline.pattern: ^\[0-9]{4}-[0-9]{2}-[0-9]{2}
   multiline.negate: false
   multiline.match: after

steffens · July 11, 2019, 11:51am

Can you share your complete configuration file? The config file is somewhat sensitive to indentation and where you put your config. Just having a snippet makes it difficult to see if the setting is at the wrong place or not

your escaping looks a little inconsistent. For regular expressions one should use single quotes '. With single quotes there is no need to escape the input string, as the YAML parser will keep strings in single quotes intact as is:

multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'

amr.abdelfattah · July 11, 2019, 12:06pm

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The filebeat.full.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

#=========================== Filebeat prospectors =============================

filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.

- input_type: log
  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/*.log
    - /app/Graylog/log/*
    #- /app/Graylog/log
    #- c:\programdata\elasticsearch\logs\*

  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  #exclude_lines: ["^DBG","^[[:space:]]<"]

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  #include_lines: ["^ERR", "^WARN","^[[:space:]]+(at|\.{3})\b|^Caused by"]

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #exclude_files: [".gz$"]

  # Optional additional fields. These field can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  #  level: debug
  #  review: 1

  ### Multiline options

  # Mutiline can be used for log messages spanning multiple lines. This is common
  # for Java Stack Traces or C-Line Continuation

  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [

  multiline.pattern: ^\[0-9]{4}-[0-9]{2}-[0-9]{2}

  # Defines if the pattern set under pattern should be negated or not. Default is false.
  multiline.negate: false

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
  # that was (not) matched before or after or as long as a pattern is not matched based on negate.
  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
  multiline.match: after


#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging

#================================ Outputs =====================================

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.

#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["10.208.221.18:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["10.208.221.18:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: critical, error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]

amr.abdelfattah · July 11, 2019, 12:11pm

Also,I used single quote as per your advise ,but the issue still exist as per attached screenshot

steffens · July 12, 2019, 1:12pm

Checking the table here: https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html#multiline

The table suggests to set negate: true and match: after in your multiline configuration.

Hartfordfive did create the filebeat-multiline-tester for users to test multiline settings from the command line. I'd recommend to give this a try.

amr.abdelfattah · July 15, 2019, 10:48am

Working now .Thanks steffens!

amr.abdelfattah · July 16, 2019, 2:17pm

Hi @steffens,
I want to use my multiline pattern to filter out all not needed messages and get only Error messages
(i.e message that contains Error word) ,any suggestions what i can put in below regular expression:

^[0-9]{4}-[0-9]{2}-[0-9]{2}

steffens · July 17, 2019, 11:32am

You can't filter out lines using multiline, as multiline checks consecutive lines.
Filtering out intermediate lines could break overall output.

But you can filter out events after the multiline filter using the exclude_lines or include_lines setting. This guarantees that the events stay intact and are parseable (if required).

I don't see any log level in your logs. To me it sounds like you want to filter errors vs. info/debug messages. In this case do not filter based on the actual error messages if possible, as this might become pretty fragile and a wrong filter can remove the wrong logs. Is there a chance to output the log level in your application?

system · August 14, 2019, 1:33pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Filebeat parsing issue Beats filebeat	1	357	March 4, 2020
Multiline FIlebeat configuration for custom log Beats filebeat	3	2305	March 17, 2017
Filebeat multiline regular expression Beats filebeat	1	306	October 25, 2019
SOLVED - Multi Line Help Beats filebeat	2	378	May 14, 2019
Multiline is not fetching desired result from filebeat to ES Beats filebeat	4	355	March 17, 2020

Filebeat configuration for multi line option (regex required)

Related topics