# Filebeat modules - problem with pipeline.json **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [October 9, 2017, 8:03am UTC](https://discuss.elastic.co/t/filebeat-modules-problem-with-pipeline-json/103190 "2017-10-09T08:03:20Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Miroslav\_Kudlac](https://avatars.discourse-cdn.com/v4/letter/m/e274bd/32.png) [@Miroslav\_Kudlac](https://discuss.elastic.co/u/Miroslav_Kudlac)\ **Post date:** [October 9, 2017, 8:03am UTC](https://discuss.elastic.co/t/filebeat-modules-problem-with-pipeline-json/103190/1 "2017-10-09T08:03:20Z")
Hi, I am trying to make my own module, but I have a problem right on the beginning. I tried to write my own pipeline.json file, but after command execution I get this error: > filebeat -e --modules test -setup > 2017/10/09 07:43:43.391721 beat.go:297: INFO Home path: [C:\Users\miroslav.kudlac\Desktop\filebeat-5.6.2-windows-x86\_64] Config path: [C:\Users\miroslav.kudlac\Desktop\filebeat-5.6.2-windows-x86\_64] Data path: [C:\Users\miroslav.kudlac\Desktop\filebeat-5.6.2-windows-x86\_64\data] Logs path: [C:\Users\miroslav.kudlac\Desktop\filebeat-5.6.2-windows-x86\_64\logs] > 2017/10/09 07:43:43.391721 metrics.go:23: INFO Metrics logging every 30s > 2017/10/09 07:43:43.392722 beat.go:192: INFO Setup Beat: filebeat; Version: 5.6.2 > 2017/10/09 07:43:43.393725 output.go:258: INFO Loading template enabled. Reading template file: C:\Users\miroslav.kudlac\Desktop\filebeat-5.6.2-windows-x86\_64\filebeat.template.json > 2017/10/09 07:43:43.395723 output.go:269: INFO Loading template enabled for Elasticsearch 2.x. Reading template file: C:\Users\miroslav.kudlac\Desktop\filebeat-5.6.2-windows-x86\_64\filebeat.template-es2x.json > 2017/10/09 07:43:43.398722 output.go:281: INFO Loading template enabled for Elasticsearch 6.x. Reading template file: C:\Users\miroslav.kudlac\Desktop\filebeat-5.6.2-windows-x86\_64\filebeat.template-es6x.json > 2017/10/09 07:43:43.400721 client.go:128: INFO Elasticsearch url: [http://localhost:9200](http://localhost:9200) > 2017/10/09 07:43:43.401720 outputs.go:108: INFO Activated elasticsearch as output plugin. > 2017/10/09 07:43:43.402720 publish.go:300: INFO Publisher name: FESK-LTP0089 > 2017/10/09 07:43:43.418726 async.go:63: INFO Flush Interval set to: 1s > 2017/10/09 07:43:43.418726 async.go:64: INFO Max Bulk Size set to: 50 > 2017/10/09 07:43:43.457247 filebeat.go:46: INFO Enabled modules/filesets: test (error) > 2017/10/09 07:43:43.491243 client.go:128: INFO Elasticsearch url: [http://localhost:9200](http://localhost:9200) > 2017/10/09 07:43:43.519241 client.go:667: INFO Connected to Elasticsearch version 5.6.2 > 2017/10/09 07:44:13.395145 metrics.go:39: INFO Non-zero metrics in the last 30s: libbeat.es.publish.read\_bytes=433 libbeat.es.publish.write\_bytes=321 > 2017/10/09 07:44:43.408291 metrics.go:34: INFO No non-zero metrics in the last 30s > 2017/10/09 07:44:58.425241 beat.go:331: INFO Kibana dashboards successfully loaded. > 2017/10/09 07:44:58.426238 client.go:128: INFO Elasticsearch url: [http://localhost:9200](http://localhost:9200) > 2017/10/09 07:44:58.443242 client.go:667: INFO Connected to Elasticsearch version 5.6.2 > 2017/10/09 07:44:58.446742 beat.go:233: INFO filebeat start running. > 2017/10/09 07:44:58.447243 client.go:128: INFO Elasticsearch url: [http://localhost:9200](http://localhost:9200) > 2017/10/09 07:44:58.466247 client.go:667: INFO Connected to Elasticsearch version 5.6.2 > 2017/10/09 07:44:58.474251 metrics.go:51: INFO Total non-zero values: libbeat.es.publish.read\_bytes=28358 libbeat.es.publish.write\_bytes=141868 > 2017/10/09 07:44:58.474251 metrics.go:52: INFO Uptime: 1m15.0875224s > 2017/10/09 07:44:58.475264 beat.go:237: INFO filebeat stopped. > 2017/10/09 07:44:58.475264 beat.go:346: CRIT Exiting: Error loading pipeline for fileset test/error: couldn't load pipeline: couldn't load json. Error: 400 Bad Request. Response body: {"error":{"root\_cause":[{"type":"parse\_exception","reason":"[processors] required property is missing","header":{"property\_name":"processors"}}],"type":"parse\_exception","reason":"[processors] required property is missing","header":{"property\_name":"processors"}},"status":400} > Exiting: Error loading pipeline for fileset test/error: couldn't load pipeline: couldn't load json. Error: 400 Bad Request. Response body: {"error":{"root\_cause":[{"type":"parse\_exception","reason":"[processors] required property is missing","header":{"property\_name":"processors"}}],"type":"parse\_exception","reason":"[processors] required property is missing","header":{"property\_name":"processors"}},"status":400} which is trully strange, because when I tried to run pipeline from restclient, everything went fine: [POST] [http://localhost:9200/\_ingest/pipeline/\_simulate](http://localhost:9200/_ingest/pipeline/_simulate) { "pipeline": { "description": "Pipeline for parsing MySQL slow logs.", "processors": [ { "grok": { "field": "message", "patterns": [ "%{DATA:time} %{NUMBER:thread} \[%{DATA:log\_level}\] %{GREEDYDATA:log\_message}" ] } } ] }, "docs": [ { "\_source": { "message": "2017-09-26T13:10:01.850659Z 0 [Note] Plugin mysqlx reported: 'Server starts handling incoming connections'" } } ] } RESPONSE: {"docs":[{"doc":{"\_index":"\_index","\_type":"\_type","\_id":"\_id","\_source":{"log\_level":"Note","log\_message":"Plugin mysqlx reported: 'Server starts handling incoming connections'","time":"2017-09-26T13:10:01.850659Z","thread":"0","message":"2017-09-26T13:10:01.850659Z 0 [Note] Plugin mysqlx reported: 'Server starts handling incoming connections'"},"\_ingest":{"timestamp":"2017-10-09T07:37:36.353Z"}}}]} my pipeline.json file looks like: { "description": "Pipeline for parsing MySQL error logs.", "pipeline": { "processors": [ { "grok": { "field": "message", "patterns": [ "%{DATA:time} %{NUMBER:thread} \[%{DATA:log\_level}\] %{GREEDYDATA:log\_message}" ], "ignore\_missing": false } } ] } } I have downloaded filebeat-5.6.2-windows-x86\_64. Also what should I do in next step, if I want to configure what indices and dashboards/searches/visualizations should be created after I run my test module... I am reading about import\_dashboards, i see there is some import\_dashboard.exe file, and here [https://www.elastic.co/guide/en/beats/libbeat/current/import-dashboards.html](https://www.elastic.co/guide/en/beats/libbeat/current/import-dashboards.html) is some manual to import some dashboards... question is how to link module in the way he knows what index should be created or what files should be imported via import\_dashboard... ---
**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\ **Post date:** [October 9, 2017, 3:25pm UTC](https://discuss.elastic.co/t/filebeat-modules-problem-with-pipeline-json/103190/2 "2017-10-09T15:25:55Z")
Could you share your `pipeline.json` you are trying to send using Filebeat? There are a few differences between the way Filebeat sends pipelines to ES and Simulate API. For example you must write `\\[%DATA:log_level}\\]`. But according to the error messages there might be further problems. ---
**Author:** ![Miroslav\_Kudlac](https://avatars.discourse-cdn.com/v4/letter/m/e274bd/32.png) [@Miroslav\_Kudlac](https://discuss.elastic.co/u/Miroslav_Kudlac)\ **Post date:** [October 9, 2017, 6:13pm UTC](https://discuss.elastic.co/t/filebeat-modules-problem-with-pipeline-json/103190/3 "2017-10-09T18:13:00Z")
You Can see the exact context od my pipeline.json in the Topic. ---
**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\ **Post date:** [October 9, 2017, 8:48pm UTC](https://discuss.elastic.co/t/filebeat-modules-problem-with-pipeline-json/103190/4 "2017-10-09T20:48:54Z")
For me your pipeline works properly after changing `\` to `\\`. Here is the content of `pipeline.json` I ingest into ES using Filebeat: ```auto { "description": "Pipeline for parsing MySQL slow logs.", "processors": [ { "grok": { "field": "message", "patterns": [ "%{DATA:time} %{NUMBER:thread} \\[%{DATA:log_level}\\] %{GREEDYDATA:log_message}" ] } } ] } ``` Does it work for you? Is it what you have in your `pipeline.json`? ---
**Author:** ![Miroslav\_Kudlac](https://avatars.discourse-cdn.com/v4/letter/m/e274bd/32.png) [@Miroslav\_Kudlac](https://discuss.elastic.co/u/Miroslav_Kudlac)\ **Post date:** [October 10, 2017, 7:32am UTC](https://discuss.elastic.co/t/filebeat-modules-problem-with-pipeline-json/103190/5 "2017-10-10T07:32:21Z")
yes, my pipeline.json is the same, editor just didnt show one . `{ "pipeline": { "processors": [{ "grok": { "field": "message", "patterns": [ "%{DATA:time} %{NUMBER:thread} \\[%{DATA:log_level}\\] %{GREEDYDATA:log_message}" ] } } ] } }` ---
**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\ **Post date:** [October 10, 2017, 8:53am UTC](https://discuss.elastic.co/t/filebeat-modules-problem-with-pipeline-json/103190/6 "2017-10-10T08:53:17Z")
Your JSON is different than the one above. It should not be under key `pipeline`. The key `processors` must be on the top level of keys, as seen in my version. That's why your log contain `{"type":"parse_exception","reason":"[processors] required property is missing","header":{"property_name":"processors"}`. ---
**Author:** ![Miroslav\_Kudlac](https://avatars.discourse-cdn.com/v4/letter/m/e274bd/32.png) [@Miroslav\_Kudlac](https://discuss.elastic.co/u/Miroslav_Kudlac)\ **Post date:** [October 10, 2017, 10:07am UTC](https://discuss.elastic.co/t/filebeat-modules-problem-with-pipeline-json/103190/7 "2017-10-10T10:07:39Z")
YES!! great thanks... Now how to create my own index -\> not use default filebeat index. Also how to create my own dashboards via input\_dashboard. Also I have a problem that one line of log is written to index twice, once parsed based on pipeline.json and once unparsed message ---
**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\ **Post date:** [October 10, 2017, 11:25am UTC](https://discuss.elastic.co/t/filebeat-modules-problem-with-pipeline-json/103190/8 "2017-10-10T11:25:49Z")
To define index pattern for you new module, you must edit `fields.yml` under folder `_meta`. After you finished creating `fields.yml`, you should run `make update` to generate Dashboards are created manually in Kibana. After you are done with your dashboard, you can export it using `python dev-tools/export_dashboards.py`. There is an extensive guide on creating Filebeat modules: [https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-modules-devguide.html](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-modules-devguide.html) Regarding log duplication, do you mean that your message show up once in Kibana, but when you see the details of the message, there is an additional field called `message`? If yes, you can discard it by adding a remove processor to your pipeline: ```auto { "remove": { "field": "message" } } ``` ---
**Author:** ![Miroslav\_Kudlac](https://avatars.discourse-cdn.com/v4/letter/m/e274bd/32.png) [@Miroslav\_Kudlac](https://discuss.elastic.co/u/Miroslav_Kudlac)\ **Post date:** [October 10, 2017, 12:33pm UTC](https://discuss.elastic.co/t/filebeat-modules-problem-with-pipeline-json/103190/9 "2017-10-10T12:33:48Z")
No, two documents in the index are created... And I am not aware that i have two parsers somewhere... But I will check it. Thanks for now. Also make update is not working as Makefile is not existing in default downloadable package for windows ---
**Author:** ![Miroslav\_Kudlac](https://avatars.discourse-cdn.com/v4/letter/m/e274bd/32.png) [@Miroslav\_Kudlac](https://discuss.elastic.co/u/Miroslav_Kudlac)\ **Post date:** [October 16, 2017, 1:16pm UTC](https://discuss.elastic.co/t/filebeat-modules-problem-with-pipeline-json/103190/10 "2017-10-16T13:16:48Z")
Is it possible to create for each module new index? Or I can only change default pattern (filebeat) to something else. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 13, 2017, 1:17pm UTC](https://discuss.elastic.co/t/filebeat-modules-problem-with-pipeline-json/103190/11 "2017-11-13T13:17:02Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.