Unable to remove extra fields when using filebeat

Elastic Stack Kibana
1

so i had a very simple ELK install with a very simple filebeat install as well

elk works fine, however with filebeat i get a bunch of fields that i do not want like

  • nginx.error.message
  • mongodb.log.message

and a bunch of others

I cant seem to get rid of them
tried using include_fields and drop_fields and it doesnt seem to be working

below is the msg in elk

{
"@timestamp": [
"2025-11-21T08:12:17.894Z"
],
"auditd.log.msg": [
"Fri Nov 21 08:12:17 UTC 2025 - Test log entry from bash script #8"
],
"icinga.debug.message": [
"Fri Nov 21 08:12:17 UTC 2025 - Test log entry from bash script #8"
],
"icinga.main.message": [
"Fri Nov 21 08:12:17 UTC 2025 - Test log entry from bash script #8"
],
"icinga.startup.message": [
"Fri Nov 21 08:12:17 UTC 2025 - Test log entry from bash script #8"
],
"logstash.log.message": [
"Fri Nov 21 08:12:17 UTC 2025 - Test log entry from bash script #8"
],
"message": [
"Fri Nov 21 08:12:17 UTC 2025 - Test log entry from bash script #8"
],
"mongodb.log.message": [
"Fri Nov 21 08:12:17 UTC 2025 - Test log entry from bash script #8"
],
"mysql.error.message": [
"Fri Nov 21 08:12:17 UTC 2025 - Test log entry from bash script #8"
],
"nginx.error.message": [
"Fri Nov 21 08:12:17 UTC 2025 - Test log entry from bash script #8"
],
"postgresql.log.message": [
"Fri Nov 21 08:12:17 UTC 2025 - Test log entry from bash script #8"
],
"redis.log.message": [
"Fri Nov 21 08:12:17 UTC 2025 - Test log entry from bash script #8"
],
"system.auth.message": [
"Fri Nov 21 08:12:17 UTC 2025 - Test log entry from bash script #8"
],
"system.auth.timestamp": [
"2025-11-21T08:12:17.894Z"
],
"system.syslog.message": [
"Fri Nov 21 08:12:17 UTC 2025 - Test log entry from bash script #8"
],
"system.syslog.timestamp": [
"2025-11-21T08:12:17.894Z"
],
"_id": "qtZ4pZoBMci39k2wyhsg",
"_index": ".ds-testz-2025.11.21-000001",
"_score": null
}

here is the config for filebeat

```

filebeat.inputs:

# Debug logs
- type: log
  enabled: true
  paths:
    - /home/admin/test/*
  fields:
    log_type: debug
    source_type: debug_files
  fields_under_root: false


# ============================== Filebeat modules ===============================

# ================================== General ===================================
name: "ip-10-20-11-174"
tags: ["debian", "filebeat"]

# ================================== Outputs ===================================
output.elasticsearch:
  hosts: ["10.20.11.174:9200"]
  protocol: "https"
  username: "elastic"
  password: "${ELASTIC_PASSWORD}"

  # SSL settings
  ssl:
    enabled: true
    verification_mode: none  # Use 'full' in production with proper certs

#  # Index settings
  index: "test"
#
#  # Force regular indices instead of data streams
#  allow_older_versions: true

# ================================= Templates ===================================
setup.template.name: "test"
setup.template.pattern: "test"
setup.template.settings:
  index:
    number_of_shards: 1
    number_of_replicas: 0
setup.template.mappings:
  properties:
    "@timestamp":
      type: date
    message:
      type: text
    fields:
      properties:
        log_type:
          type: keyword
        source_type:
          type: keyword

## ================================= Processors =================================
processors:
  - drop_fields:
      fields:
        - "nginx.error.message"
  - include_fields:
      fields:
        - "@*"        # Include all @timestamp fields
        - "message"  # Include message and message.keyword

feel like there’s something very basic im missing.

2

so it looks like when the index is of type data stream it adds those extra fields

when i use logs as the type it doesnt