Hiding or Excluding Fields in Filebeat Data Ingestion via MQTT

Hi,

I'm encountering some challenges while ingesting data using Filebeat with MQTT. My setup involves ingesting data from multiple MQTT topics that follow this naming structure:

<ABBREVIATION>/<CITY_NAME>/<SIGNAL_CODE>

From these MQTT topics, I successfully extract two key values - CITY_NAME AND SIGNAL_CODE. I extracted these using the dissect processor to use them to construct the target data stream name. For example, the city name is converted to lowercase and combined with the signal code to form the data stream name where the data is indexed.

While this works as intended, I don't like the fact that the CITY_NAME and SIGNAL_CODE parameters are represented as fields in any way (mapped or unmapped) and seen in Kibana dashboard. There are two reasons I want to avoid storing or exposing these fields in Kibana and these are:

  • Redundancy for Customers - Our customers use the Kibana dashboard, and these values are redundant for them. Displaying them may cause confusion or just unnecessary clutter.
  • Using unnecessary space - Storing these additional fields (whether mapped or not) will consume unnecessary disk space, which I would like to avoid.

My Question

How can I ensure that the values of these fields:

  1. Are not stored in Elasticsearch (to save space), and/or
  2. Are hidden or excluded from being displayed in the Kibana dashboard?

Here is my current Filebeat configuration:

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

filebeat.inputs:
- type: mqtt
  enabled: true
  hosts: 
    - <mqtt source>
  username: <username>
  password: <password>
  topics: 
    - "<ABBREVIATION>/<CITY_NAME>/<SIGNAL_CODE>"
    - "<ABBREVIATION>/<CITY_NAME>/<SIGNAL_CODE>"
    - "<ABBREVIATION>/<CITY_NAME>/<SIGNAL_CODE>"

  ssl:
    enabled: true
    verification_mode: "none"

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

processors:
- dissect:
    tokenizer: "ABBREVIATION/%{city_name}/%{signal_code}"
    field: "mqtt.topic"
    target_prefix: ""
- script:
    lang: javascript
    id: extract_and_clean
    source: >
        function process(event) {
		    // Save signal code as variable to be used as field name
            var signalName = event.Get("signal_code");
            var message = JSON.parse(event.Get("message"));

            // Extract relevant fields
            event.Put("signal_id", message.signalID);
            event.Put("quality", message.quality);
            
            // Add field with signal name key with signal value
            event.Put(signalName, message.value);

            // Remove unnecessary fields
            var fieldsToRemove = [
                "message", "mqtt", "agent", "host", "ecs", "input"
            ]; 
            fieldsToRemove.forEach(function(field) {
                event.Delete(field);
            });
        }


cloud.id: "<CLOUD_ID>"
cloud.auth: "<CLOUD_AUTH>"

output.elasticsearch:
  enabled: true
  index: "anomaly-detection-%{[city_name]}-%{[signal_code]}"

setup.template:
  name: "anomaly-detection"
  pattern: "anomaly-detection-*"

logging.level: info
logging.selectors: ["*"]

Filebeat version: 8.15.1

Please let me know if you need any additional details or context!

Best regards,

Tadej Boncelj