Multiline Pattern - Filebeat

Elastic Stack Beats
1

Hello, I am having problems making a multiline match. The logs come from a file saved on the client machine. I send to the logstash server with filebeat. In the filebeat configuration I have the following:

multiline.pattern: '^%{LOGLEVEL} %{TIMESTAMP_ISO8601} \['
multiline.negate: true
multiline.match: after

The log file looks like this:

ERROR 2019-09-13 08:27:24,794 [pool-2-thread-17604] com.mirth.connect.server.userutil.MessageObject: The messageObject.getRawData() method is deprecated and will soon be removed. Please use connectorMessage.getRawData() instead.
ERROR 2019-09-13 08:27:24,794 [pool-2-thread-17604] com.mirth.connect.server.userutil.ResponseFactory: The getSuccessResponse(message) method is deprecated and will soon be removed. Please use getSentResponse(message) instead.
ERROR 2019-09-13 08:27:47,515 [Timer-40] com.mirth.connect.connectors.jdbc.DatabaseReceiver: Failed to poll for messages from the database in channel
com.mirth.connect.connectors.jdbc.DatabaseReceiverException: Unrecognized value returned from script in channel expected ResultSet or List<Map<String, Object>>: org.mozilla.javascript.Undefined@76ea45d7
	at com.mirth.connect.connectors.jdbc.DatabaseReceiverScript.poll(DatabaseReceiverScript.java:112)
	at com.mirth.connect.connectors.jdbc.DatabaseReceiver.poll(DatabaseReceiver.java:111)
	at com.mirth.connect.donkey.server.channel.PollConnector$PollConnectorTask.run(PollConnector.java:141)
	at java.util.TimerThread.mainLoop(Unknown Source)
	at java.util.TimerThread.run(Unknown Source)
ERROR 2019-09-13 08:27:47,906 [Timer-47] com.mirth.connect.connectors.jdbc.DatabaseReceiver: Failed to poll for messages from the database in channel "Tal"
com.mirth.connect.connectors.jdbc.DatabaseReceiverException: Unrecognized value returned from script in channel "Tal", expected ResultSet or List<Map<String, Object>>: org.mozilla.javascript.Undefined@76ea45d7
	at com.mirth.connect.connectors.jdbc.DatabaseReceiverScript.poll(DatabaseReceiverScript.java:112)
	at com.mirth.connect.connectors.jdbc.DatabaseReceiver.poll(DatabaseReceiver.java:111)
	at com.mirth.connect.donkey.server.channel.PollConnector$PollConnectorTask.run(PollConnector.java:141)
	at java.util.TimerThread.mainLoop(Unknown Source)
	at java.util.TimerThread.run(Unknown Source)
ERROR 2019-09-13 08:28:05,267 [pool-2-thread-17603] com.mirth.connect.server.userutil.SerializerFactory: The getHL7Serializer() method is deprecated and will soon be removed. Please use the "Convert HL7 v2.x" templates from the References tab instead. Look at the tooltips to see the available property keys. The new method will strip namespaces by default unless the 'stripNamespaces' property is set to false.
ERROR 2019-09-13 08:28:12,424 [pool-2-thread-17604] com.mirth.connect.server.userutil.SerializerFactory: The getHL7Serializer() method is deprecated and will soon be removed. Please use the "Convert HL7 v2.x" templates from the References tab instead. Look at the tooltips to see the available property keys. The new method will strip namespaces by default unless the 'stripNamespaces' property is set to false.
ERROR 2019-09-13 08:28:14,049 [pool-2-thread-17603] transformer: TypeError: Cannot read property "CD47y" from undefined
ERROR 2019-09-13 08:28:14,143 [pool-2-thread-17603] com.mirth.connect.server.userutil.ResponseFactory: The getSuccessResponse(message) method is deprecated and will soon be removed. Please use getSentResponse(message) instead.
ERROR 2019-09-13 08:28:14,940 [pool-2-thread-17603] com.mirth.connect.server.controllers.DonkeyEngineController: Could not find channel to route to: 1cf4c80f-3cfc-4686-8e93-8b39b2d6b537
com.mirth.connect.donkey.server.channel.ChannelException
	at com.mirth.connect.server.controllers.DonkeyEngineController.dispatchRawMessage(DonkeyEngineController.java:511)
	at com.mirth.connect.server.userutil.VMRouter.routeMessageByChannelId(VMRouter.java:154)
	at com.mirth.connect.server.userutil.VMRouter.routeMessageByChannelId(VMRouter.java:139)
	at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)

When de logs arribe in logstash they match in a message and not in different ones for eachline that begins with the "error" and the content below.

Any idea?? Thx

2

@gerard.ramos,

Please try the below pattern in multiline.pattern

multiline.pattern: '[A-Z]{5} [0-9]{4}-[0-9]{2}-[0-9]{2}'

Thanks.

3

Heey Tek, thanks for your reply. I'm still getting the same output.

4

@gerard.ramos, Have you restarted the filebeat service after making the changes? If yes please provide the configuration file of filebeat.

Thanks.

5

Yees i restarted the filebeat service, this is my configuration file:

filebeat.inputs:
    - type: log
      enabled: true
      paths:
        - C:\Program Files\Mirth Connect\logs\mirth*  
      exclude_files: ['.log$']
      multiline.pattern: '[A-Z]{5} [0-9]{4}[0-9]{2}-[0-9]{2}' 
      multiline.negate: true
      multiline.match: after
    filebeat.config.modules: 
      path: ${path.config}/modules.d/*.yml 
      reload.enabled: false 
    setup.template.settings:
      index.number_of_shards: 1
    setup.kibana:
    output.logstash:
      hosts: ["192.168.1.76:5443"]
    processors:
      - add_host_metadata: ~
      - add_cloud_metadata: ~
6

@gerard.ramos,

filebeat.inputs:
    - type: log
      enabled: true
      paths:
        - C:\Program Files\Mirth Connect\logs\*  
      exclude_files: ['.log$']
      multiline.pattern: '[A-Z]{5} [0-9]{4}[0-9]{2}-[0-9]{2}' 
      multiline.negate: true
      multiline.match: after
    filebeat.config.modules: 
      path: ${path.config}/modules.d/*.yml 
      reload.enabled: false 
    setup.template.settings:
      index.number_of_shards: 1
    setup.kibana:
    output.logstash:
      hosts: ["192.168.1.76:5443"]
    processors:
      - add_host_metadata: ~
      - add_cloud_metadata: ~

In you above configuration its seems that your are excluding the files with .log extension. What is the extension of your files which you are trying to parse? If they have .log extension please remove the line exclude_files from your config.

And use my config i have made small change.
Thanks.

7

If I am excluded messages and generating my own test. The problem is not that they do not arrive but that they are not separated into different messages.

8

Hello again tek, I have already found the error, in the pattern that you had passed me a - like this:

multiline.pattern: '[A-Z]{5} [0-9]{4}[0-9]{2}-[0-9]{2}' 
multiline.pattern: '[A-Z]{5} [0-9]{4}-[0-9]{2}-[0-9]{2}'

Thanks for the replys!!

9

@gerard.ramos

is it working now? That was typo mistake. Now i have edited my post.

Thanks.

10

Yes is working, thank you.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.