Hi,
I'm encountering some challenges while ingesting data using Filebeat with MQTT. My setup involves ingesting data from multiple MQTT topics that follow this naming structure:
<ABBREVIATION>/<CITY_NAME>/<SIGNAL_CODE>
From these MQTT topics, I successfully extract two key values - CITY_NAME
AND SIGNAL_CODE
. I extracted these using the dissect
processor to use them to construct the target data stream name. For example, the city name is converted to lowercase and combined with the signal code to form the data stream name where the data is indexed.
While this works as intended, I don't like the fact that the CITY_NAME
and SIGNAL_CODE
parameters are represented as fields in any way (mapped or unmapped) and seen in Kibana dashboard. There are two reasons I want to avoid storing or exposing these fields in Kibana and these are:
- Redundancy for Customers - Our customers use the Kibana dashboard, and these values are redundant for them. Displaying them may cause confusion or just unnecessary clutter.
- Using unnecessary space - Storing these additional fields (whether mapped or not) will consume unnecessary disk space, which I would like to avoid.
My Question
How can I ensure that the values of these fields:
- Are not stored in Elasticsearch (to save space), and/or
- Are hidden or excluded from being displayed in the Kibana dashboard?
Here is my current Filebeat configuration:
filebeat.config:
modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
filebeat.inputs:
- type: mqtt
enabled: true
hosts:
- <mqtt source>
username: <username>
password: <password>
topics:
- "<ABBREVIATION>/<CITY_NAME>/<SIGNAL_CODE>"
- "<ABBREVIATION>/<CITY_NAME>/<SIGNAL_CODE>"
- "<ABBREVIATION>/<CITY_NAME>/<SIGNAL_CODE>"
ssl:
enabled: true
verification_mode: "none"
filebeat.autodiscover:
providers:
- type: docker
hints.enabled: true
processors:
- dissect:
tokenizer: "ABBREVIATION/%{city_name}/%{signal_code}"
field: "mqtt.topic"
target_prefix: ""
- script:
lang: javascript
id: extract_and_clean
source: >
function process(event) {
// Save signal code as variable to be used as field name
var signalName = event.Get("signal_code");
var message = JSON.parse(event.Get("message"));
// Extract relevant fields
event.Put("signal_id", message.signalID);
event.Put("quality", message.quality);
// Add field with signal name key with signal value
event.Put(signalName, message.value);
// Remove unnecessary fields
var fieldsToRemove = [
"message", "mqtt", "agent", "host", "ecs", "input"
];
fieldsToRemove.forEach(function(field) {
event.Delete(field);
});
}
cloud.id: "<CLOUD_ID>"
cloud.auth: "<CLOUD_AUTH>"
output.elasticsearch:
enabled: true
index: "anomaly-detection-%{[city_name]}-%{[signal_code]}"
setup.template:
name: "anomaly-detection"
pattern: "anomaly-detection-*"
logging.level: info
logging.selectors: ["*"]
Filebeat version: 8.15.1
Please let me know if you need any additional details or context!
Best regards,
Tadej Boncelj