Problems getting pipeline to run in filebeat module

Elastic Stack Beats
1

hi folks
I'm having some trouble with GROK within a custom filebeat module
Runing filebeat 6.5 on windows

The mudule is ingesting & sending the logs to elasticserach without any issues
so i dont know if the module is calling the ingest pipeline correctly, debug on filebeat didnt show any obvious problems.

I've listed the contenst of my manifest.yml & the pipeline.json files.
i'm setting a new field in pipeline.json ifthe processor completes or fails but when i look at the record in elasticsearch the grok_status feild is missing & no groking has taken place

manifest.yml
module_version: 1.0

var:
  - name: paths
default:
  - //non-prod-smb-ext.onxinas1a.bskyb.com/N09-KanaResponse/KanaShare_DEV/logs/*.xml

ingest_pipeline: ingest/pipeline.json
input: config/log.yml

pipeline.json

{
    "description": "Pipeline for parsing Kana application logs",
    "processors": [
        {
            "grok": {
                "field": "message",
                "patterns": [
                    "%{INT:kananode} %{DATA:parition} %{INT:kanaint}"
					],
				"ignore_missing": true
            },
			"set": {
                "field": "grok_status",
                "value": "GROK PARSE OK"
            }
  }
    ],
    "on_failure": [
        {
            "set": {
                "field": "grok_status",
                "value": "GROK PARSE ERROR"
            }
        }
    ]
}
2

@John_Swift Can you check if your pipeline is installed on Elasticsearch?

3

& just like that it all clicks into place.....
Thanks Pier, thats exsaclty what i was missing !!!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.