No custom module fields in Kibana

[mvasilenko (Mykhailo Vasylenko)]

Hello,

I'm interested in creating custom Filebeat module, to parse our app log,
so I've followed this guide
https://www.elastic.co/elasticon/conf/2018/sf/build-your-own-filebeat-module,
simulated pipeline, grok patterns wokring fine,
but no custom fields shows in kibana, what I'm missing here?

i'm using filebeat autodiscover feature for docker logs,
and other modules, like nginx, working fine, but not mine custom one.

filebeat.autodiscover:
  providers:
   - type: docker
     templates:
       - condition.or:
           - contains.docker.container.image: "myimage"
         config:
           - module: mymodule
             access:
               prospector:
                 type: docker
                 containers.stream: stdout
                 containers.ids:
                   - "${data.docker.container.id}"
         error:
               prospector:
                 type: docker
                 containers.stream: stderr
                 containers.ids:
                   - "${data.docker.container.id}"

[pierhugues (Pier-Hugues Pellerin)]

@mvasilenko Did you try using your module outside the autodiscover context using the log input instead of the docker input?

[mvasilenko (Mykhailo Vasylenko)]

@pierhugues thank you for the answer, i've messed with stderr/stdout,
but the question remains, what exact actions do i need to perform after generating/editing
custom module fields, to load it into kibana/elasticsearch?

is it sufficient to update /etc/filebeat/fields.yml and /usr/share/filebeat/module/mymodule?

i'm using ubuntu 16

[pierhugues (Pier-Hugues Pellerin)]

It should be sufficient to do so.

I would still use this guide To make sure you can easily test your new module.

[mvasilenko (Mykhailo Vasylenko)]

@pierhugues ok, i've tested module by adding pipeline to ES

curl -H 'Content-Type: application/json' -XPUT localhost:9200/_ingest/pipeline/filebeat-6.3.0-mymodule-backend -d@/usr/share/filebeat/module/mymodule/backend/ingest/pipeline.json

and simulating in kibana dev tools

POST _ingest/pipeline/filebeat-6.3.0-mymodule-backend/_simulate

{
  "docs" : [
    { "_source": {
        "message": "[2018-07-02 21:06:57 +0000] [INFO] message"} }
  ]
}

the output looks good, now my idea is to combine docker autodiscovery feature with my custom module for specific containers, few questions arise

• do i need to specify anything in filebeat.yml pipeline section? or it is already applied at autodiscovery/mymodule level

• what files need to be copied to remote hosts, in addition to standard filebeat package?

[mvasilenko (Mykhailo Vasylenko)]

Update:

looks like adding filebeat.overwrite_pipelines: true
to /etc/filebeat/filebeat.yml fixes this issue

another thing i can add - you must to test pipeline like this, not via ES pipeline simulate api

~/go/src/github.com/elastic/beats/filebeat $ ./scripts/tester/tester -elasticsearch https://elk:9200 \
-pipeline /usr/share/filebeat/module/mymodule/backend/ingest/pipeline.json \
-logfile sample.log --simulate.verbose --verbose

Now I'm looking to adopt this model for logging in k8s, any hints?
Thank you