Hi Everyone,
I'm working on creating a new module for Filebeat to capture Greenplum command logs. After creating a module and fileset and adding in a first pipeline, the test_reload_writes_pipeline test fails with error code -15. I am unsure of what that error code means.
An awkward thing about my pipeline is that it has double quotes which I use a slash to cancel, I was able to run the system tests fine on a commit where the pipeline was a copy of the postgres pipeline, which makes me believe that something about how I wrote my pipeline.json file is causing the problem. Here is my pipeline
{
"description": "Pipeline for parsing Greenplum logs.",
"processors": [
{
"grok": {
"field": "message",
"ignore_missing": true,
"patterns": [
"^%{DATETIME:greenplum.log.timestamp},,,%{WORD:greenplum.log.process_id},th-%{WORD:greenplum.log.thread_id},,,%{DATETIME:greenplum.log.timestamp},%{NUMBER:greenplum.log.transaction_id},,,%{WORD:greenplum.log.gp_segment.type}-%{WORD:greenplum.log.gp_segment.id},,,,,"%{WORD:greenplum.log.level}","%{NUMBER:greenplum.log.state_code}","%{GREEDYDATA:greenplum.log.message}",,,,,,,%{NUMBER:greenplum.log.cursor_position},,"%{GREEDYDATA:greenplum.log.file.name}",%{NUMBER:greenplum.log.file.line},"
],
"pattern_definitions": {
"DATETIME": "[-0-9]+ %{TIME} %{WORD:event.timezone}",
"GREEDYDATA": "(.|\n|\t)",
"GREENPLUM_DB_NAME": "[a-zA-Z0-9_]+[a-zA-Z0-9_\$]",
"GREENPLUM_QUERY_STEP": "%{WORD:greenplum.log.query_step}(?: | %{WORD:greenplum.log.query_name})?"
}
}
},
{
"date": {
"field": "greenplum.log.timestamp",
"target_field": "@timestamp",
"formats": [
"yyyy-MM-dd HH:mm:ss.SSS zz", "yyyy-MM-dd HH:mm:ss zz"
]
}
}, {
"script": {
"lang": "painless",
"source": "ctx.event.duration = Math.round(ctx.temp.duration * params.scale)",
"params": { "scale": 1000000 },
"if": "ctx.temp?.duration != null"
}
}, {
"remove": {
"field": "temp.duration",
"ignore_missing": true
}
}
],
"on_failure": [
{
"set": {
"field": "error.message",
"value": "{{ _ingest.on_failure_message }}"
}
}
]
}
I have forked the beats repo in case anyone wants to see something else in the project structure.
Please let me know if you have any ideas about what I need to fix to pass the system test.
Thanks,
Christian
