Multiple prospectors not working under filebeat

Elastic Stack Beats
1

Hi,

I am using filebeat 5.4.0 with below configuration, however multiline is somehow not woking.

filebeat.prospectors:
- input_type: log
  paths:
    - '/var/logs/JINK/*.log'
  tags: [jinklogs]

  multiline.pattern: '^JINK'
  multiline.negate: true
  multiline.match: after

- input_type: log
  paths:
    - '/root/sample/*.log'
  tags: [applogs]

  multiline.pattern: '\d+\/\d+\/\d+ \d+:\d+:\d+'
  multiline.negate: true
  multiline.match: after


processors:
- drop_fields.fields: [offset]
- drop_fields.fields: [beat.version]

  ### Multiline options

  # Mutiline can be used for log messages spanning multiple lines. This is common
  # for Java Stack Traces or C-Line Continuation
#----------------------------- Console output ---------------------------------
output.console:
  # Boolean flag to enable or disable the output module.
  enabled: true

  # Pretty print json event
  pretty: true

For now i am testing by printing to console to validate o/p before I set output to logstash.

My log files are as below:

for applogs

9/9/2016 2:20:57 AM|Command Line Params:
D:\APP\CT\V1.0\CT.exe
/APPLICATION_ID=10
/MAC_SERVICE_URL=tcp://localhost:80/Service
/FREQUENCY=10
/PASS_STRING=Data Source=10.1.1.1\CT;Initial USER=ctuser;User ID=#12868#;Password=#xxxxxxx#
LOGCONFIG

9/9/2016 2:20:56 AM: Start
Unable to find plugin: D:\CT\Extensions\CTP.dll

for jinklogs

JINK|Error|0|08-25-2016 07:39:09:45|[JINK]|01OBN|2||0|SessionInit|Unable to sign-in|1
JINK|Critical|0|08-25-2016 07:39:11:53|[JINK||||0|AuthToken|*** Exception ***
Msg: exception thrown by generateToken()
ExceptionMsg:System.ApplicationException: generateToken() - not able to generate token for ID : 12221 , appid : 1ASPSD , UTC Time : 2016-08-25 07:39:09 , refer : <a href='https://webapp.jink.local.do' >https://webservices.jink.local/intranet/hr.do</a>
   in d:\JINK\Userdata\JinkApp.cs:line 16|1
JINK|Verbose|0|08-25-2016 07:39:11:53|[JINK]||||0|AuthToken|GenerateKeyEnd|1
JINK|Critical|0|08-25-2016 07:39:11:53|[JINK]||||0|Configuration|*** Exception ***
Msg: Unhandle Exception
ExceptionMsg:System.ApplicationException: generateToken() - not able to generate token for ID : 091S , appid : 2PSRD , UTC Time : 2016-08-25 07:40:29 , refer : <a href='https://webapp.jink.local.do' >https://webservices.jink.local/intranet/hr.do</a>
   in d:\JINK\Userdata\JinkApp.cs:line 216|2
JINK|Information|0|08-25-2016 07:42:09:45|[JINK]|01O9N|2||0|SessionInit|Sign-in success|1

Any ideas what could be the issue ?

2

Have you tried the tester that is linked to from this page? https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html

3

I tested the patterns against your log samples. They look to be working?

{
  "@timestamp": "2017-05-25T12:36:36.702Z",
  "beat": {
    "hostname": "host",
    "name": "host",
    "version": "5.4.0"
  },  
  "input_type": "log",
  "message": "JINK|Critical|0|08-25-2016 07:39:11:53|[JINK]||||0|Configuration|*** Exception ***\nMsg: Unhandle Exception\nExceptionMsg:System.ApplicationException: generateToken() - not able to generate token for ID : 091S , appid : 2PSRD , UTC Time : 2016-08-25 07:40:29 , refer : \u003ca href='https://webapp.jink.local.do' \u003ehttps://webservices.jink.local/intranet/hr.do\u003c/a\u003e\n   in d:\\JINK\\Userdata\\JinkApp.cs:line 216|2",
  "offset": 980,
  "source": "/Users/akroh/go/src/github.com/elastic/beats/filebeat/.test/multiline/jinklogs.log",
  "tags": [
    "jinklogs"
  ],  
  "type": "log"
}
{
  "@timestamp": "2017-05-25T12:36:36.702Z",
  "beat": {
    "hostname": "host",
    "name": "host",
    "version": "5.4.0"
  },  
  "input_type": "log",
  "message": "9/9/2016 2:20:57 AM|Command Line Params:\nD:\\APP\\CT\\V1.0\\CT.exe\n/APPLICATION_ID=10\n/MAC_SERVICE_URL=tcp://localhost:80/Service\n/FREQUENCY=10\n/PASS_STRING=Data Source=10.1.1.1\\CT;Initial USER=ctuser;User ID=#12868#;Password=#xxxxxxx#\nLOGCONFIG\n",
  "offset": 243,
  "source": "/Users/akroh/go/src/github.com/elastic/beats/filebeat/.test/multiline/applogs.log",
  "tags": [
    "applogs"
  ],  
  "type": "log"
}
4

It works if I have only one prospector with multiline codec however when I have more than one prospectors with each having multiline, none work. Am I doing something wrong here or is it expected behavior ?

5

the multiline reader is per prospector. Having multiple prosepectors each with multiline should work.

Kind of sounds something bogus in your config file. Have you tried with each prospector type, one after another?

6

Yes works Steffen. Thanks

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.