# New Module Fails System Pipeline Reload Test **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [January 20, 2020, 8:05pm UTC](https://discuss.elastic.co/t/new-module-fails-system-pipeline-reload-test/215793 "2020-01-20T20:05:44Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![seaseao](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/seaseao/32/48107_2.png) [@seaseao](https://discuss.elastic.co/u/seaseao)\ **Post date:** [January 20, 2020, 8:05pm UTC](https://discuss.elastic.co/t/new-module-fails-system-pipeline-reload-test/215793/1 "2020-01-20T20:05:44Z")
Hi Everyone, I'm working on creating a new module for Filebeat to capture Greenplum command logs. After creating a module and fileset and adding in a first pipeline, the test\_reload\_writes\_pipeline test fails with error code -15. I am unsure of what that error code means. An awkward thing about my pipeline is that it has double quotes which I use a slash to cancel, I was able to run the system tests fine on a commit where the pipeline was a copy of the postgres pipeline, which makes me believe that something about how I wrote my pipeline.json file is causing the problem. Here is my pipeline > { > "description": "Pipeline for parsing Greenplum logs.", > "processors": [ > { > "grok": { > "field": "message", > "ignore\_missing": true, > "patterns": [ > "^%{DATETIME:greenplum.log.timestamp},,,%{WORD:greenplum.log.process\_id},th-%{WORD:greenplum.log.thread\_id},,,%{DATETIME:greenplum.log.timestamp},%{NUMBER:greenplum.log.transaction\_id},,,%{WORD:greenplum.log.gp\_segment.type}-%{WORD:greenplum.log.gp\_segment.id},,,,,"%{WORD:greenplum.log.level}","%{NUMBER:greenplum.log.state\_code}","%{GREEDYDATA:greenplum.log.message}",,,,,,,%{NUMBER:greenplum.log.cursor\_position},,"%{GREEDYDATA:greenplum.log.file.name}",%{NUMBER:greenplum.log.file.line}," > ], > "pattern\_definitions": { > "DATETIME": "[-0-9]+ %{TIME} %{WORD:event.timezone}", > "GREEDYDATA": "(.|\n|\t)_", > "GREENPLUM\_DB\_NAME": "[a-zA-Z0-9\_]+[a-zA-Z0-9\_\$]_", > "GREENPLUM\_QUERY\_STEP": "%{WORD:greenplum.log.query\_step}(?: | %{WORD:greenplum.log.query\_name})?" > } > } > }, > { > "date": { > "field": "greenplum.log.timestamp", > "target\_field": "@timestamp", > "formats": [ > "yyyy-MM-dd HH:mm:ss.SSS zz", "yyyy-MM-dd HH:mm:ss zz" > ] > } > }, { > "script": { > "lang": "painless", > "source": "ctx.event.duration = Math.round(ctx.temp.duration \* params.scale)", > "params": { "scale": 1000000 }, > "if": "ctx.temp?.duration != null" > } > }, { > "remove": { > "field": "temp.duration", > "ignore\_missing": true > } > } > ], > "on\_failure": [ > { > "set": { > "field": "error.message", > "value": "{{ \_ingest.on\_failure\_message }}" > } > } > ] > } I have [forked the beats repo](https://github.com/seaseao/beats/tree/master/filebeat/module/greenplum) in case anyone wants to see something else in the project structure. Please let me know if you have any ideas about what I need to fix to pass the system test. Thanks, Christian ---
**Author:** ![seaseao](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/seaseao/32/48107_2.png) [@seaseao](https://discuss.elastic.co/u/seaseao)\ **Post date:** [January 21, 2020, 10:29pm UTC](https://discuss.elastic.co/t/new-module-fails-system-pipeline-reload-test/215793/2 "2020-01-21T22:29:20Z")
So I got the module to pass the testsuite after I made a change where I removed pattern definitions that weren't used by the pipeline. "GREENPLUM\_DB\_NAME": "[a-zA-Z0-9\_]+[a-zA-Z0-9\_\$]\*", "GREENPLUM\_QUERY\_STEP": "%{WORD:greenplum.log.query\_step}(?: | %{WORD:greenplum.log.query\_name})?" When I got rid of those, the pipeline reload test behaved better. I also encountered some weird inconsistencies (to me) where the order I ran parts of the testsuite in mattered. If I ran system-tests-environment before integration-tests-environment, the system tests would hang on test\_fileset\_file\_068\_auditd. When I ran the testsuite after running the system tests, the reload test failed. However I restarted docker and ran the testsuite and found success. If anyone thinks this weirdness can be attributed to me making a blunder with respect to docker or venv please let me know. Also if this is the wrong forum to bring up making new modules let me know and I'll stop spamming it. Github for those interested: > **[seaseao/beats](https://github.com/seaseao/beats/tree/master/filebeat/module/greenplum)** > > :tropical\_fish: Beats - Lightweight shippers for Elasticsearch & Logstash - seaseao/beats Thanks, Christian ---
**Author:** ![Mario\_Castro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mario_castro/32/35107_2.png) [@Mario\_Castro](https://discuss.elastic.co/u/Mario_Castro)\ **Post date:** [January 23, 2020, 6:59pm UTC](https://discuss.elastic.co/t/new-module-fails-system-pipeline-reload-test/215793/3 "2020-01-23T18:59:08Z")
Hi @seaseao 🙂 I strongly recommend you to open a PR to the main beats repo. The reason for this is that developing filebeat modules is a bit tricky sometimes and it's always better to open PR's early so we can help you with the entire process, including the build and test system. No pressure here, we are simply happy to help and it's always easier to help in Github with actual code when talking about developing new modules. Don't feel discouraged because "this is not finished yet, I have to polish it, etc." Best regards ---
**Author:** ![seaseao](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/seaseao/32/48107_2.png) [@seaseao](https://discuss.elastic.co/u/seaseao)\ **Post date:** [January 23, 2020, 8:57pm UTC](https://discuss.elastic.co/t/new-module-fails-system-pipeline-reload-test/215793/4 "2020-01-23T20:57:17Z")
Hi Mario! I've opened an initial PR here [https://github.com/elastic/beats/pull/15794](https://github.com/elastic/beats/pull/15794) I'm excited to keep working on the feature, it's cool learning a couple new languages at once! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 20, 2020, 8:57pm UTC](https://discuss.elastic.co/t/new-module-fails-system-pipeline-reload-test/215793/5 "2020-02-20T20:57:22Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.