Help on showing Custom File beat Module Fields in Kibana

[CoreyF (Corey)]

Hello, I have created a custom File Beat Module using Creating a New Beat.
The data is making it to Elasticsearch and I can see it under the “Discover” Tab in Kibana. In Kibana -> Visualize -> Visual Builder -> Time Series I can see the “Count” for the number of records in the index I created with the Custom File Beat. But I cannot see my custom Fields.

Filebeat.yml Config:

#=========================== Filebeat inputs =============================
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /ixialogs/*_WebStatic.log
#============================= Filebeat modules ===============================
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
#==================== Elasticsearch template setting ==========================
setup.template.settings:
  index.number_of_shards: 3
#================================ General =====================================
fields:
   service: WebStatic
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  hosts: ["100.68.2.101:9200"]
  index: "ixia-%{+yyyy}"
  pipeline: WebStatic
#================================ Procesors =====================================
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
#================================ Logging =====================================
setup.template.name: "filebeat-%{[beat.version]}"
setup.template.pattern: "filebeat-%{[beat.version]}-*"

Screen Shot of Discover Tab showing records and fields:

Screen Shot of Visualize showing records in index:

Screen Shot of Fields not showing:

What is my next step to get the Custom Fields to show up in Visualize Graphs?
Do I need to do the Create a New Dashboard section or did I miss something?

Thanks for the help!
Corey

[steffens (Steffen Siering)]

Do you have a module definition?

If you have a module collecting the data, then do not configure the input and do not configure output.elasticsearch.pipeline. It is the module which has this knowledge and configures those.

The field not being shown in the listing looks like a kibana issue. I See you are using the time series visual builder. Have you tried other visualizations (e.g. normal line graph)?

Also double check the kibana index pattern for filebeat under the Management tab. The way your filebeat configuration looks like I'm not sure a module is really used. W

Without module and extra fields definitions of the IXIA fields you are missing the field names and types from the template mapping (that is Elasticsearch auto-derives the types for these unknown fields) and you are missing the field names in the kibana index pattern. If that's the case you can try to refresh the index pattern in the kibana Management UI, so kibana internally learns about these fields.

[CoreyF (Corey)]

Hello, You were correct. I did not have the module loading correctly. I have seen done some re-configuration and testing. Now I believe the Custom Module is loading correctly.

Here is the new "/etc/filebeat/filebeat.yml" file.

filebeat.inputs:
- type: log
  enabled: false
  paths:
    - /var/log/*.log
filebeat.config.modules:
  enabled: true
  path: ${path.config}/modules.d/*.yml
reload.enabled: false
filebeat.modules:
- module: ixia
setup.template.settings:
  index.number_of_shards: 3
setup.kibana:
  host: "100.68.2.101:5601"
output.elasticsearch:
  hosts: ["100.68.2.101:9200"]
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

But I'm still having the same problem.

@steffens

Have you tried other visualizations (e.g. normal line graph)?

Yes, I have tried the other visualizations such as line graphs.

Above you mention:

@steffens

(that is Elasticsearch auto-derives the types for these unknown fields)

So does that mean there is nothing else that I'm missing in the config setup to create the custom fields to show up in the Kibana Visualization graphs?

Yes, I have refreshed the index pattern in the Kibana Management UI with no luck.

Is there any debug log files that would help?

Thank you....Corey

[steffens (Steffen Siering)]

For logging see: https://www.elastic.co/guide/en/beats/filebeat/current/configuration-logging.html

If you run filebeat in debug mode (start with -d '*' or set log level to debug and set selectors to ["*"] in config file), you will see each event in the log to be published by filebeat.

I think you are missing indentation on reload.enabled: false . Unrelated to your issue, I just noticed.