Hello, I have created a custom File Beat Module using Creating a New Beat.
The data is making it to Elasticsearch and I can see it under the “Discover” Tab in Kibana. In Kibana -> Visualize -> Visual Builder -> Time Series I can see the “Count” for the number of records in the index I created with the Custom File Beat. But I cannot see my custom Fields.
Screen Shot of Discover Tab showing records and fields:
Screen Shot of Visualize showing records in index:
Screen Shot of Fields not showing:
What is my next step to get the Custom Fields to show up in Visualize Graphs?
Do I need to do the Create a New Dashboard section or did I miss something?
If you have a module collecting the data, then do not configure the input and do not configure output.elasticsearch.pipeline. It is the module which has this knowledge and configures those.
The field not being shown in the listing looks like a kibana issue. I See you are using the time series visual builder. Have you tried other visualizations (e.g. normal line graph)?
Also double check the kibana index pattern for filebeat under the Management tab. The way your filebeat configuration looks like I'm not sure a module is really used. W
Without module and extra fields definitions of the IXIA fields you are missing the field names and types from the template mapping (that is Elasticsearch auto-derives the types for these unknown fields) and you are missing the field names in the kibana index pattern. If that's the case you can try to refresh the index pattern in the kibana Management UI, so kibana internally learns about these fields.
Hello, You were correct. I did not have the module loading correctly. I have seen done some re-configuration and testing. Now I believe the Custom Module is loading correctly.
Here is the new "/etc/filebeat/filebeat.yml" file.
Yes, I have tried the other visualizations such as line graphs.
Above you mention:
So does that mean there is nothing else that I'm missing in the config setup to create the custom fields to show up in the Kibana Visualization graphs?
Yes, I have refreshed the index pattern in the Kibana Management UI with no luck.
If you run filebeat in debug mode (start with -d '*' or set log level to debug and set selectors to ["*"] in config file), you will see each event in the log to be published by filebeat.
I think you are missing indentation on reload.enabled: false . Unrelated to your issue, I just noticed.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.