Multiple problems with Kibana Dashboards (Metricbeat/Winlogbeat)

Hey Everyone. I have followed a basic implementation of ELK Stack (version 7.10) on a Windows 10 machine. Now though, I'm having problems with the visualisation of data from two Beats agents (Metricbeat and Winlogbeat). The Beats agents are running on the same device that runs the ELK server (I've also tested running the agents on another Win10 machine on the same LAN, the problem is still there).

I have followed the official quick start install for both agents :

I have been successful with retrieving data from the "Discover" panel in Kibana :

However, I have not been able at all to view any data in most of the different loaded Dashboards, whether for Metricbeat or Winlogbeat. These are the Dashboards that I've decided to keep :

For example, if I go in the "[Metricbeat Windows] Services ECS" Dashboard, I get only blank panels with the error

Saved field "windows.service.id" is invalid for use with the "Unique Count" aggregation. Please select a new field.

Except for one panel which features the error

Saved field "windows.service.display_name" is invalid for use with the "Terms" aggregation. Please select a new field.

Unfortunately, I have many errors in other dashboards, even for those that have successfully retrieved certain information :

The only dashboards where I've found a fair amount of sucess are the "Overview" Dashboards :

I'm using Beats to send to Logstash (Yes, I have manually indexed the templates to Elasticsearch on setup). For Metricbeat, only the "System" and "Windows" modules are enabled and configured.

Here are my config files (with pertinent info) :

  • metricbeat.yml :
# =========================== Modules configuration ============================

metricbeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "https://<IP@>:5601"
  username: "metricbeat_internal"
  password: <pwd>
  #ssl.enabled: true

# ------------------------------ Logstash Output -------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["<IP@>:5045"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  ssl.certificate_authorities: ["certs/ca.crt"]

  # Certificate for SSL client authentication
  #ssl.certificate: 

  # Client Certificate Key
  #ssl.key: 
  username: "metricbeat_internal"
  password: <pwd>
# # ================================= Processors =================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~


# ================================== Logging ===================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
logging.level: debug
logging.to_files: true

# ================================= ILM ======================================
setup.ilm.check_exists: false
  • winlogbeat.yml :
# ======================== Winlogbeat specific options =========================

# event_logs specifies a list of event logs to monitor as well as any
# accompanying options. The YAML data type of event_logs is a list of
# dictionaries.
#
# The supported keys are name (required), tags, fields, fields_under_root,
# forwarded, ignore_older, level, event_id, provider, and include_xml. Please
# visit the documentation for the complete details of each option.
# https://go.es.io/WinlogbeatConfig

winlogbeat.event_logs:
  - name: Application
    ignore_older: 72h

  - name: System

  - name: Security
    processors:
      - script:
          lang: javascript
          id: security
          file: ${path.home}/module/security/config/winlogbeat-security.js

  - name: Microsoft-Windows-Sysmon/Operational
    processors:
      - script:
          lang: javascript
          id: sysmon
          file: ${path.home}/module/sysmon/config/winlogbeat-sysmon.js

  - name: Windows PowerShell
    event_id: 400, 403, 600, 800
    processors:
      - script:
          lang: javascript
          id: powershell
          file: ${path.home}/module/powershell/config/winlogbeat-powershell.js

  - name: Microsoft-Windows-PowerShell/Operational
    event_id: 4103, 4104, 4105, 4106
    processors:
      - script:
          lang: javascript
          id: powershell
          file: ${path.home}/module/powershell/config/winlogbeat-powershell.js

  - name: ForwardedEvents
    tags: [forwarded]
    processors:
      - script:
          when.equals.winlog.channel: Security
          lang: javascript
          id: security
          file: ${path.home}/module/security/config/winlogbeat-security.js
      - script:
          when.equals.winlog.channel: Microsoft-Windows-Sysmon/Operational
          lang: javascript
          id: sysmon
          file: ${path.home}/module/sysmon/config/winlogbeat-sysmon.js
      - script:
          when.equals.winlog.channel: Windows PowerShell
          lang: javascript
          id: powershell
          file: ${path.home}/module/powershell/config/winlogbeat-powershell.js
      - script:
          when.equals.winlog.channel: Microsoft-Windows-PowerShell/Operational
          lang: javascript
          id: powershell
          file: ${path.home}/module/powershell/config/winlogbeat-powershell.js
  
  - name: HardwareEvents
  - name: HP Sure Start  

# ====================== Elasticsearch template settings =======================

setup.template.settings:
  index.number_of_shards: 1

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "https://IP@:5601"
  username: "winlogbeat_internal"
  password: <pwd>

# ------------------------------ Logstash Output -------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["IP@:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  ssl.certificate_authorities: ["certs/ca.crt"]

  # Certificate for SSL client authentication
  username: "winlogbeat_internal"
  password: <pwd>

# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~

# ================================= ILM ======================================
setup.ilm.enabled: auto
setup.ilm.rollover_alias: "winlogbeat"
setup.ilm.check_exists: false
setup.ilm.overwrite: true
  • metricbeat/modules.d/system.yml :
# Module: system
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/7.10/metricbeat-module-system.html

- module: system
  period: 10s
  metricsets:
    - cpu
    #- load
    - memory
    #- network
    - process
    - process_summary
    - socket_summary
    #- entropy
    #- core
    - diskio
    #- socket       
    #- service
    #- users
  process.include_top_n:
    by_cpu: 5      # include top 5 processes by CPU
    by_memory: 5   # include top 5 processes by memory

- module: system
  period: 1m
  metricsets:
    - filesystem
    - fsstat
  processors:
  - drop_event.when.regexp:
      system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)'

- module: system
  period: 15m
  metricsets:
    - uptime
  • metricbeat/modules.d/windows.yml
# Module: windows
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/7.10/metricbeat-module-windows.html

- module: windows
  metricsets:
    - service
  enabled: true
  period: 1m

- module: windows
  metricsets:
    - perfmon
  enabled: false
  period: 10s
  perfmon.ignore_non_existent_counters: false
  perfmon.group_measurements_by_instance: false
  perfmon.queries:
    - object: "Process"
      instance: ["svchost*", "conhost*"]
      counters:
      - name: "% Processor Time"
        field: time.processor.pct
        format: "float"
      - name: "Thread Count"
        field: thread_count
      - name: "IO Read Operations/sec"
    - object: "PhysicalDisk"
      field : "disk"
      instance: "*"
      counters:
      - name: "Disk Writes/sec"
      - name: "% Disk Write Time"
        field: "write_time"
        format: "float"
  • The metricbeat pipeline :
input {
  beats {
    port => 5045
	ssl => true
    ssl_key => "certs/logstash.pkcs8.key"
    ssl_certificate => "certs/logstash.crt"
  }
}

output {
  elasticsearch {
    hosts => ["https://IP@:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}"
	cacert => "certs/ca.crt"
    user => "logstash_internal"
    password => "<pwd>"
  }
}
  • The winlogbeat pipeline :
input {
  beats {
    port => 5044
	ssl => true
    ssl_key => "certs/logstash.pkcs8.key"
    ssl_certificate => "certs/logstash.crt"
  }
}

output {
  elasticsearch {
    hosts => ["https://IP@:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}"
	cacert => "certs/ca.crt"
    user => "logstash_internal"
    password => "<pwd>"
  }
}

I think something went wrong in this step - most of the problems I can see in your screenshots are very likely caused by fields being of type text instead of type keyword. E.g. host.name - if beats are creating the mapping themselves, they will specify host.name as type keyword. However if you just pipe the data into an index with auto-mapping enabled, it will make host.name of type text and create a separate field host.name.keyword which contains the keyword-indexed version of the same field.

To make the dashboard work, you have to make sure to specify the mapping before ingesting data, otherwise Elasticsearch will pick a default which is incompatible with the dashboards.

One way to clean up:

  • Delete all existing mappings
  • Point the beats directly to Elasticsearch and start them so they can set up the mapping the right way
  • Stop the beats and change the config to pipe the data through Logstash
  • As the mapping is defined in the right way now, Elasticsearch should ingest the data fine from now on
  • Dashboards should simply start to work, if they don't delete them and recreate them

Sorry, but still didn't work

Can you share the mapping of your indices?

  • Mappings for Winlogbeat index :
{
  "winlogbeat-7.10.0" : {
    "mappings" : {
      "_meta" : {
        "beat" : "winlogbeat",
        "version" : "7.10.0"
      },
      "dynamic_templates" : [
        {
          "labels" : {
            "path_match" : "labels.*",
            "match_mapping_type" : "string",
            "mapping" : {
              "type" : "keyword"
            }
          }
        },
        {
          "container.labels" : {
            "path_match" : "container.labels.*",
            "match_mapping_type" : "string",
            "mapping" : {
              "type" : "keyword"
            }
          }
        },
        {
          "fields" : {
            "path_match" : "fields.*",
            "match_mapping_type" : "string",
            "mapping" : {
              "type" : "keyword"
            }
          }
        },
        {
          "docker.container.labels" : {
            "path_match" : "docker.container.labels.*",
            "match_mapping_type" : "string",
            "mapping" : {
              "type" : "keyword"
            }
          }
        },
        {
          "kubernetes.labels.*" : {
            "path_match" : "kubernetes.labels.*",
            "mapping" : {
              "type" : "keyword"
            }
          }
        },
        {
          "kubernetes.annotations.*" : {
            "path_match" : "kubernetes.annotations.*",
            "mapping" : {
              "type" : "keyword"
            }
          }
        },
        {
          "winlog.event_data" : {
            "path_match" : "winlog.event_data.*",
            "match_mapping_type" : "string",
            "mapping" : {
              "type" : "keyword"
            }
          }
        },
        {
          "winlog.user_data" : {
            "path_match" : "winlog.user_data.*",
            "match_mapping_type" : "string",
            "mapping" : {
              "type" : "keyword"
            }
          }
        },
        {
          "strings_as_keyword" : {
            "match_mapping_type" : "string",
            "mapping" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            }
          }
        }
      ],
      "date_detection" : false,
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "@version" : {
          "type" : "keyword",
          "ignore_above" : 1024
        },
        "agent" : {
          "properties" : {
            "build" : {
              "properties" : {
                "original" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                }
              }
            },
            "ephemeral_id" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "hostname" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "id" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "name" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "type" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "version" : {
              "type" : "keyword",
              "ignore_above" : 1024
            }
          }
        },
        "as" : {
          "properties" : {
            "number" : {
              "type" : "long"
            },
            "organization" : {
              "properties" : {
                "name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024,
                  "fields" : {
                    "text" : {
                      "type" : "text",
                      "norms" : false
                    }
                  }
                }
              }
            }
          }
        },
        "client" : {
          "properties" : {
            "address" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "as" : {
              "properties" : {
                "number" : {
                  "type" : "long"
                },
                "organization" : {
                  "properties" : {
                    "name" : {
                      "type" : "keyword",
                      "ignore_above" : 1024,
                      "fields" : {
                        "text" : {
                          "type" : "text",
                          "norms" : false
                        }
                      }
                    }
                  }
                }
              }
            },
            "bytes" : {
              "type" : "long"
            },
            "domain" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
       ......
 "geo" : {
              "properties" : {
                "city_name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "continent_name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "country_iso_code" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "country_name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "location" : {
                  "type" : "geo_point"
                },
                "name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "region_iso_code" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "region_name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                }
              }
            },
            "ip" : {
              "type" : "ip"
            },
            "mac" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "nat" : {
              "properties" : {
                "ip" : {
                  "type" : "ip"
                },
                "port" : {
                  "type" : "long"
                }
              }
            },
            "packets" : {
              "type" : "long"
            },
            "port" : {
              "type" : "long"
            },
            "registered_domain" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "top_level_domain" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "user" : {
              "properties" : {
                "domain" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "email" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "full_name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024,
                  "fields" : {
                    "text" : {
                      "type" : "text",
                      "norms" : false
                    }
                  }
                },
                "group" : {
                  "properties" : {
                    "domain" : {
                      "type" : "keyword",
                      "ignore_above" : 1024
                    },
                    "id" : {
                      "type" : "keyword",
                      "ignore_above" : 1024
                    },
                    "name" : {
                      "type" : "keyword",
                      "ignore_above" : 1024
                    }
                  }
                },
                "hash" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "id" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024,
                  "fields" : {
                    "text" : {
                      "type" : "text",
                      "norms" : false
                    }
                  }
                },
                "roles" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                }
              }
            }
          }
        },