Filebeat modules - problem with pipeline.json

Elastic Stack Beats
1

Hi, I am trying to make my own module, but I have a problem right on the beginning. I tried to write my own pipeline.json file, but after command execution I get this error:

filebeat -e --modules test -setup
2017/10/09 07:43:43.391721 beat.go:297: INFO Home path: [C:\Users\miroslav.kudlac\Desktop\filebeat-5.6.2-windows-x86_64] Config path: [C:\Users\miroslav.kudlac\Desktop\filebeat-5.6.2-windows-x86_64] Data path: [C:\Users\miroslav.kudlac\Desktop\filebeat-5.6.2-windows-x86_64\data] Logs path: [C:\Users\miroslav.kudlac\Desktop\filebeat-5.6.2-windows-x86_64\logs]
2017/10/09 07:43:43.391721 metrics.go:23: INFO Metrics logging every 30s
2017/10/09 07:43:43.392722 beat.go:192: INFO Setup Beat: filebeat; Version: 5.6.2
2017/10/09 07:43:43.393725 output.go:258: INFO Loading template enabled. Reading template file: C:\Users\miroslav.kudlac\Desktop\filebeat-5.6.2-windows-x86_64\filebeat.template.json
2017/10/09 07:43:43.395723 output.go:269: INFO Loading template enabled for Elasticsearch 2.x. Reading template file: C:\Users\miroslav.kudlac\Desktop\filebeat-5.6.2-windows-x86_64\filebeat.template-es2x.json
2017/10/09 07:43:43.398722 output.go:281: INFO Loading template enabled for Elasticsearch 6.x. Reading template file: C:\Users\miroslav.kudlac\Desktop\filebeat-5.6.2-windows-x86_64\filebeat.template-es6x.json
2017/10/09 07:43:43.400721 client.go:128: INFO Elasticsearch url: http://localhost:9200
2017/10/09 07:43:43.401720 outputs.go:108: INFO Activated elasticsearch as output plugin.
2017/10/09 07:43:43.402720 publish.go:300: INFO Publisher name: FESK-LTP0089
2017/10/09 07:43:43.418726 async.go:63: INFO Flush Interval set to: 1s
2017/10/09 07:43:43.418726 async.go:64: INFO Max Bulk Size set to: 50
2017/10/09 07:43:43.457247 filebeat.go:46: INFO Enabled modules/filesets: test (error)
2017/10/09 07:43:43.491243 client.go:128: INFO Elasticsearch url: http://localhost:9200
2017/10/09 07:43:43.519241 client.go:667: INFO Connected to Elasticsearch version 5.6.2
2017/10/09 07:44:13.395145 metrics.go:39: INFO Non-zero metrics in the last 30s: libbeat.es.publish.read_bytes=433 libbeat.es.publish.write_bytes=321
2017/10/09 07:44:43.408291 metrics.go:34: INFO No non-zero metrics in the last 30s
2017/10/09 07:44:58.425241 beat.go:331: INFO Kibana dashboards successfully loaded.
2017/10/09 07:44:58.426238 client.go:128: INFO Elasticsearch url: http://localhost:9200
2017/10/09 07:44:58.443242 client.go:667: INFO Connected to Elasticsearch version 5.6.2
2017/10/09 07:44:58.446742 beat.go:233: INFO filebeat start running.
2017/10/09 07:44:58.447243 client.go:128: INFO Elasticsearch url: http://localhost:9200
2017/10/09 07:44:58.466247 client.go:667: INFO Connected to Elasticsearch version 5.6.2
2017/10/09 07:44:58.474251 metrics.go:51: INFO Total non-zero values: libbeat.es.publish.read_bytes=28358 libbeat.es.publish.write_bytes=141868
2017/10/09 07:44:58.474251 metrics.go:52: INFO Uptime: 1m15.0875224s
2017/10/09 07:44:58.475264 beat.go:237: INFO filebeat stopped.
2017/10/09 07:44:58.475264 beat.go:346: CRIT Exiting: Error loading pipeline for fileset test/error: couldn't load pipeline: couldn't load json. Error: 400 Bad Request. Response body: {"error":{"root_cause":[{"type":"parse_exception","reason":"[processors] required property is missing","header":{"property_name":"processors"}}],"type":"parse_exception","reason":"[processors] required property is missing","header":{"property_name":"processors"}},"status":400}
Exiting: Error loading pipeline for fileset test/error: couldn't load pipeline: couldn't load json. Error: 400 Bad Request. Response body: {"error":{"root_cause":[{"type":"parse_exception","reason":"[processors] required property is missing","header":{"property_name":"processors"}}],"type":"parse_exception","reason":"[processors] required property is missing","header":{"property_name":"processors"}},"status":400}

which is trully strange, because when I tried to run pipeline from restclient, everything went fine:

[POST] http://localhost:9200/_ingest/pipeline/_simulate
{
"pipeline": {
"description": "Pipeline for parsing MySQL slow logs.",
"processors": [
{
"grok": {
"field": "message",
"patterns": [
"%{DATA:time} %{NUMBER:thread} \[%{DATA:log_level}\] %{GREEDYDATA:log_message}"
]
}
}
]
},
"docs": [
{
"_source": {
"message": "2017-09-26T13:10:01.850659Z 0 [Note] Plugin mysqlx reported: 'Server starts handling incoming connections'"
}
}
]
}

RESPONSE:

{"docs":[{"doc":{"_index":"_index","_type":"_type","_id":"_id","_source":{"log_level":"Note","log_message":"Plugin mysqlx reported: 'Server starts handling incoming connections'","time":"2017-09-26T13:10:01.850659Z","thread":"0","message":"2017-09-26T13:10:01.850659Z 0 [Note] Plugin mysqlx reported: 'Server starts handling incoming connections'"},"_ingest":{"timestamp":"2017-10-09T07:37:36.353Z"}}}]}

my pipeline.json file looks like:
{
"description": "Pipeline for parsing MySQL error logs.",
"pipeline": {
"processors": [
{
"grok": {
"field": "message",
"patterns": [
"%{DATA:time} %{NUMBER:thread} \[%{DATA:log_level}\] %{GREEDYDATA:log_message}"
],
"ignore_missing": false
}
}
]
}
}

I have downloaded filebeat-5.6.2-windows-x86_64. Also what should I do in next step, if I want to configure what indices and dashboards/searches/visualizations should be created after I run my test module... I am reading about import_dashboards, i see there is some import_dashboard.exe file, and here
https://www.elastic.co/guide/en/beats/libbeat/current/import-dashboards.html is some manual to import some dashboards... question is how to link module in the way he knows what index should be created or what files should be imported via import_dashboard...

2

Could you share your pipeline.json you are trying to send using Filebeat?
There are a few differences between the way Filebeat sends pipelines to ES and Simulate API. For example you must write \\[%DATA:log_level}\\]. But according to the error messages there might be further problems.

3

You Can see the exact context od my pipeline.json in the Topic.

4

For me your pipeline works properly after changing \ to \\.

Here is the content of pipeline.json I ingest into ES using Filebeat:

{
    "description": "Pipeline for parsing MySQL slow logs.",
    "processors": [
        {
            "grok": {
                "field": "message",
                 "patterns": [
                    "%{DATA:time} %{NUMBER:thread} \\[%{DATA:log_level}\\] %{GREEDYDATA:log_message}"
                ]
            }
        }
    ]
}

Does it work for you? Is it what you have in your pipeline.json?

5

yes, my pipeline.json is the same, editor just didnt show one .

{ "pipeline": { "processors": [ { "grok": { "field": "message", "patterns": [ "%{DATA:time} %{NUMBER:thread} \\[%{DATA:log_level}\\] %{GREEDYDATA:log_message}" ] } } ] } }

6

Your JSON is different than the one above. It should not be under key pipeline. The key processors must be on the top level of keys, as seen in my version. That's why your log contain {"type":"parse_exception","reason":"[processors] required property is missing","header":{"property_name":"processors"}.

7

YES!! great thanks... Now how to create my own index -> not use default filebeat index. Also how to create my own dashboards via input_dashboard.

Also I have a problem that one line of log is written to index twice, once parsed based on pipeline.json and once unparsed message

8

To define index pattern for you new module, you must edit fields.yml under folder _meta. After you finished creating fields.yml, you should run make update to generate
Dashboards are created manually in Kibana. After you are done with your dashboard, you can export it using python dev-tools/export_dashboards.py.
There is an extensive guide on creating Filebeat modules: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-modules-devguide.html

Regarding log duplication, do you mean that your message show up once in Kibana, but when you see the details of the message, there is an additional field called message? If yes, you can discard it by adding a remove processor to your pipeline:

{
    "remove": {
        "field": "message"
     }
}
9

No, two documents in the index are created... And I am not aware that i have two parsers somewhere... But I will check it. Thanks for now.

Also make update is not working as Makefile is not existing in default downloadable package for windows

10

Is it possible to create for each module new index? Or I can only change default pattern (filebeat) to something else.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.