Logstash pluguin to march multiline logs?

gerard.ramos · June 19, 2019, 7:29am

Hello everyone, I was trying to send logs to Elasticsearch, this logs are in a file and they are multiline like this:

ERROR 2019-06-18 05:00:15,919 [pool-1-thread-147] somelongtexthere moretext moretext
some text
some text
more text
more text
some text

ERROR 2019-06-18 05:00:15,919 [pool-1-thread-474] somelongtexthere moretext moretext

I was searching for a pluguin that can match a message every time that read ^%{LOGLEVEL} %{TIMESTAMP_ISO8601}. I tried multiline pluguin but it dosn't worked for me, i think is because it needs a pattern in every line of the log. Any ideas???

Sorry for my english level and thanks for your time.

Badger · June 19, 2019, 12:56pm

file {
    path => "/home/user/foo.txt"
    sincedb_path => "/dev/null"
    start_position => beginning
    codec => multiline {
        pattern => "^%{LOGLEVEL} %{TIMESTAMP_ISO8601} "
        negate => true
        what => previous
        auto_flush_interval => 1
    }
}

gerard.ramos · June 19, 2019, 3:20pm

Thiis filter goes inseide the input? Like this:

input {
beats {
port => 5443
type => 'log'
}
stdin {
path => "C:\Program Files\Mirth Connect\logs\mirth.log*"
sincedb_path => "/dev/null"
start_position => beginning
codec => multiline {
pattern => "^%{LOGLEVEL} %{TIMESTAMP_ISO8601} "
negate => true
what => previous
auto_flush_interval => 1
}
}
}

Badger · June 19, 2019, 3:38pm

Yes, but it would be a file input, not a stdin input. And you need to use forward slash in the path option of a file input, not backslash.

gerard.ramos · July 1, 2019, 7:50am

I have modified the input file like this:

input {
beats {
port => 5443
type => 'eventlog'
}
file {
path => "C:/Program Files/Mirth Connect/logs/mirth*"
sincedb_path => "/dev/null"
start_position => beginning
codec => multiline {
pattern => "^%{LOGLEVEL} %{TIMESTAMP_ISO8601} "
negate => true
what => previous
auto_flush_interval => 1
}
}
}

Now kivana dont get logs, any idea??

Miguel1 · July 1, 2019, 8:25am

Your are using

sincedb_path => "/dev/null"

But as far as you are using Windows, at least your path looks like this, you should go for

sincedb_path => "NUL"

If not, it will not reparse the log.

gerard.ramos · July 1, 2019, 11:35am

Oh well, I misunderstood it. I thought that the path referred to the origin of the logs. Now i have and the logs arrive:
path => "/home/administrator/foo.txt"
sincedb_path => "/dev/null"

but the messages keep coming apart and the multiline codec dont work. I don't know if is important but the logstash also recive Winlogbeat messages to.

gerard.ramos · July 2, 2019, 11:27am

I raise a more complete example (the line starts at the log level). I hope you can help me.

        ERROR 2019-06-29 09:36:42,548 [pool-2-thread-975] com.mirth.connect.server.userutil.MessageObject: The messageObject.getRawData() method is deprecated and will soon be removed. Please use connectorMessage.getRawData() instead.
        ERROR 2019-06-29 09:36:42,548 [pool-2-thread-975] com.mirth.connect.server.userutil.ResponseFactory: The getSuccessResponse(message) method is deprecated and will soon be removed. Please use getSentResponse(message) instead.
        ERROR 2019-06-29 09:39:17,620 [pool-2-thread-985] transformer: TypeError: Cannot read property "CX.1" from undefined
        ERROR 2019-06-29 09:39:18,937 [pool-2-thread-975] com.mirth.connect.server.userutil.ResponseFactory: The getSuccessResponse(message) method is deprecated and will soon be removed. Please use getSentResponse(message) instead.
        ERROR 2019-06-29 09:39:19,844 [pool-2-thread-975] com.mirth.connect.server.controllers.DonkeyEngineController: Could not find channel to route to: 1cf4c805-3cfc-4676-3e93-8b39b2d6464567
        com.mirth.connect.donkey.server.channel.ChannelException
        	at com.mirth.connect.server.controllers.DonkeyEngineController.dispatchRawMessage(DonkeyEngineController.java:511)
        	at com.mirth.connect.server.userutil.VMRouter.routeMessageByChannelId(VMRouter.java:164)
        	at com.mirth.connect.server.userutil.VMRouter.routeMessageByChannelId(VMRouter.java:149)
        	at com.mirth.connect.server.transformers.JavaScriptResponseTransformer$ResponseTransformerTask.call(JavaScriptResponseTransformer.java:110)
        	at java.util.concurrent.FutureTask.run(Unknown Source)
        	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        	at java.lang.Thread.run(Unknown Source)
        ERROR 2019-06-29 09:39:19,844 [pool-2-thread-975] com.mirth.connect.server.userutil.VMRouter: Error routing message to channel id: 1cf4c80f-3cfc-4486-8e93-8b39b2d65313123b537
        com.mirth.connect.donkey.server.channel.ChannelException

Badger · July 2, 2019, 12:58pm

You have left something out. With that input, and that multiline codec I get

   "message" => "ERROR 2019-06-29 09:36:42,548 [pool-2-thread-975] com.mirth.connect.server.userutil.MessageObject: The messageObject.getRawData() method is deprecated and will soon be removed. Please use connectorMessage.getRawData() instead.",
   "message" => "ERROR 2019-06-29 09:36:42,548 [pool-2-thread-975] com.mirth.connect.server.userutil.ResponseFactory: The getSuccessResponse(message) method is deprecated and will soon be removed. Please use getSentResponse(message) instead.",
   "message" => "ERROR 2019-06-29 09:39:17,620 [pool-2-thread-985] transformer: TypeError: Cannot read property \"CX.1\" from undefined",
   "message" => "ERROR 2019-06-29 09:39:18,937 [pool-2-thread-975] com.mirth.connect.server.userutil.ResponseFactory: The getSuccessResponse(message) method is deprecated and will soon be removed. Please use getSentResponse(message) instead.",
   "message" => "ERROR 2019-06-29 09:39:19,844 [pool-2-thread-975] com.mirth.connect.server.controllers.DonkeyEngineController: Could not find channel to route to: 1cf4c805-3cfc-4676-3e93-8b39b2d6464567\ncom.mirth.connect.donkey.server.channel.ChannelException\n    at com.mirth.connect.server.controllers.DonkeyEngineController.dispatchRawMessage(DonkeyEngineController.java:511)\n    at com.mirth.connect.server.userutil.VMRouter.routeMessageByChannelId(VMRouter.java:164)\n    at com.mirth.connect.server.userutil.VMRouter.routeMessageByChannelId(VMRouter.java:149)\n    at com.mirth.connect.server.transformers.JavaScriptResponseTransformer$ResponseTransformerTask.call(JavaScriptResponseTransformer.java:110)\n    at java.util.concurrent.FutureTask.run(Unknown Source)\n    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n    at java.lang.Thread.run(Unknown Source)",

Which means the multiline codec works if the messages have the format that you say they do.

gerard.ramos · July 2, 2019, 2:44pm

I'm really trying to find my fault but I do not see it, I pass the filter again in case you see it. Can it be a filebeat problem? Can the filter fail with the logs of other servers? Thank you for your answers

input {
  beats {
    port => 5443
    type => 'eventlog'
  }
  file {
    path => "/home/user/foo.txt"
    sincedb_path => "/dev/null"
    start_position => beginning
    codec => multiline {
      pattern => "^%{LOGLEVEL} %{TIMESTAMP_ISO8601} "
      negate => true
      what => previous
      auto_flush_interval => 1
    }
  }
}

system · July 30, 2019, 2:44pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.