Filebeat inside Kubernetes setup questions

[rocketraman (Raman Gupta)]

I'm trying to use hints-based autodiscover with Filebeat 6.4.2 inside Kubernetes.

The basic setup is working (through some trials and tribulations noted in earlier messages in this forum). However, now I'm trying to enable hints-based parsing of various inputs.

The first I'm trying to setup as I figured it would be the easiest, is the kibana module. I've added the annotation co.elastic.logs/module: kibana on the kibana pods. I've confirmed filebeat is seeing this annotation -- it gets indexed in ES and is visible in the Kibana UI:

However, the message value is still the unparsed JSON -- and none of the structured values I would expect after passing through the kibana module:

I've also tried the apache2 and elasticsearch modules, with similar no-op results.

The modules seem to be enabled inside filebeat:

2018-11-01T23:34:33.758Z	INFO	beater/filebeat.go:98	Enabled modules/filesets: kibana (log), kafka (log),  (), apache2 (access, error), elasticsearch (slowlog, audit, deprecation, gc, server)

What am I missing? How do I enable this?

Regards,
Raman

[jsoriano (Jaime Soriano)]

Hi @rocketraman,

When using autodiscover you don't have to enable the modules in the filebeat configuration, they are enabled (or disabled) automatically as needed. You should be able to see something like Autodiscover starting runner: kibana in the logs when the kibana pod is detected.

Also, you need to enable hints-based autodiscover, this is not enabled by default. Check that you have hints.enabled: true in your kubernetes autodiscover configuration.

filebeat.autodiscover:
  providers:
    - type: kubernetes
      hints.enabled: true

For the pods configuration, notice that hints have to be placed in annotations, not in labels:

annotations:
  co.elastic.logs/module: kibana

Once all the configuration is fine, to continue investigating the problem check the logs. You should be able to see some messages about autodiscover only by enabling it, or some errors if it is not working. If you don't see anything you can enable autodiscover debug with -d autodiscover.

I hope this helps.

[rocketraman (Raman Gupta)]

@jsoriano

For the pods configuration, notice that hints have to be placed in annotations , not in labels:

Oops, this is what I had wrong. Thanks!