Filebeat multiline not working

Elastic Stack Beats
1

Nginx test error log

[2017/10/13 16:17:49] [error] 21169#21169: *113356 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Class 'Elasticsearch\ClientBuilder' not found in /home/work/website/cdt0-webapp.example.com/components/search/ES.php on line 18
PHP message: PHP Stack trace:
PHP message: PHP   1. {main}() /home/work/website/cdt0-webapp.example.com/web/index.php:0
PHP message: PHP   2. yii\base\Application->run() /home/work/website/cdt0-webapp.example.com/web/index.php:39
PHP message: PHP   3. yii\web\Application->handleRequest() /home/work/website/cdt0-webapp.example.com/vendor/yiisoft/yii2/base/Application.php:380
PHP message: PHP   4. yii\base\Module->runAction() /home/work/website/cdt0-webapp.example.com/vendor/yiisoft/yii2/web/Application.php:102
PHP message: PHP   5. yii\base\Controller->runAction() /home/work/website/cdt0-webapp.example.com/vendor/yiisoft/yii2/base/Module.php:523
PHP message: PHP   6. yii\base\InlineAction->runWithParams() /home/work/website/cdt0-webapp.example.com/vendor/yiisoft/yii2/base/Controller.php:156
PHP message: PHP   7. call_user_func_array:{/home/work/website/cdt0-webapp.example.com/vendor/yiisoft/yii2/base/InlineAction.php:57}() /home/work/website/cdt0-webapp.example.com/vendor/yiisoft/yii2/base/InlineAction.php:57
PHP message: PHP   8. app\modules\trades\controllers\ListController->actionIndex() /home/work/website/cdt0-webapp.example.com/vendor/yiisoft/yii2/base/InlineAction.php:57
PHP message: PHP   9. app\components\search\Search::getAggs() /home/work/website/cdt0-webapp.example.com/modules/trades/controllers/ListController.php:151
PHP message: PHP  10. app\components\search\ES::getClient() /home/work/website/cdt0-webapp.example.com/components/search/Search.php:274" while reading response header from upstream, client: 127.0.0.1, server: cdt1-passport.example.com, request: "GET /game/huoyingrenzhe-2379-20-1 HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "cdt0-webapp.example.com", referrer: "http://cdt0-webapp.example.com/"

Filebeat config

filebeat.prospectors:
 - input_type: log
   paths:
     - /tmp/test.log
 multiline.pattern: '^\['
 multiline.negate: true
 multiline.match: after
 output.logstash:
   hosts: ["localhost:5043"]

Execute filebeat , I get following result, apparently these lines are not merged.

017/10/13 10:01:03.752015 log.go:91: INFO Harvester started for file: /tmp/test.log
2017/10/13 10:01:08.752489 client.go:214: DBG  Publish: {
  "@timestamp": "2017-10-13T10:01:03.752Z",
  "beat": {
    "hostname": "tsy01",
    "name": "tsy01",
    "version": "5.6.3"
  },
  "input_type": "log",
  "message": "2017/10/13 16:17:49 [error] 21169#21169: *113356 FastCGI sent in stderr: \"PHP message: PHP Fatal error:  Class 'Elasticsearch\\ClientBuilder' not found in /home/work/website/cdt0-webapp.example.com/components/search/ES.php on line 18",
  "offset": 233,
  "source": "/tmp/test.log",
  "type": "log"
}
2017/10/13 10:01:08.752632 client.go:214: DBG  Publish: {
  "@timestamp": "2017-10-13T10:01:03.752Z",
  "beat": {
    "hostname": "tsy01",
    "name": "tsy01",
    "version": "5.6.3"
  },
  "input_type": "log",
  "message": "PHP message: PHP Stack trace:",
  "offset": 263,
  "source": "/tmp/test.log",
  "type": "log"
}
2

The multiline settings must be added to the prospector reading the log files. Seeing just a small snippet of your config, I can not tell if you've applied the settings in the correct namespace.

3

@steffens, thanks , I updated the filebeat config , please check that again.

4

Checking the formatting of your filebeat configuration I wonder, does filebeat event startup? Doesn't look like valid YAML yo me, as all indentation is kind of out of place. Check Config file format docs.

5

Hi, the format works , you can see the events published from filebeat log I posted there, the indentation might be wrong, I just removed all irrelevant comments,but it shows correct structure,the regular expression pattern is supposed to match the log,

6

copy pasting the filebeat config in http://www.yamllint.com, it complaints about the YAML being incorrect. For multiline to work the indentation should be more like:

filebeat.prospectors:
- input_type: log
  paths:
    - /tmp/test.log
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after

output.logstash:
   hosts: ["localhost:5043"]
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.