Hey Everyone. I have followed a basic implementation of ELK Stack (version 7.10) on a Windows 10 machine. Now though, I'm having problems with the visualisation of data from two Beats agents (Metricbeat and Winlogbeat). The Beats agents are running on the same device that runs the ELK server (I've also tested running the agents on another Win10 machine on the same LAN, the problem is still there).
I have followed the official quick start install for both agents :
-
Metricbeat quick start: installation and configuration | Metricbeat Reference [8.11] | Elastic
-
Metricbeat quick start: installation and configuration | Metricbeat Reference [8.11] | Elastic
I have been successful with retrieving data from the "Discover" panel in Kibana :
However, I have not been able at all to view any data in most of the different loaded Dashboards, whether for Metricbeat or Winlogbeat. These are the Dashboards that I've decided to keep :
For example, if I go in the "[Metricbeat Windows] Services ECS" Dashboard, I get only blank panels with the error
Saved field "windows.service.id" is invalid for use with the "Unique Count" aggregation. Please select a new field.
Except for one panel which features the error
Saved field "windows.service.display_name" is invalid for use with the "Terms" aggregation. Please select a new field.
Unfortunately, I have many errors in other dashboards, even for those that have successfully retrieved certain information :
-
[Metricbeat System] Overview ECS :
-
[Winlogbeat Security] User Management Events :
The only dashboards where I've found a fair amount of sucess are the "Overview" Dashboards :
-
[Metricbeat System] Host overview ECS :
-
[Winlogbeat] Overview :
I'm using Beats to send to Logstash (Yes, I have manually indexed the templates to Elasticsearch on setup). For Metricbeat, only the "System" and "Windows" modules are enabled and configured.
Here are my config files (with pertinent info) :
- metricbeat.yml :
# =========================== Modules configuration ============================
metricbeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# ======================= Elasticsearch template setting =======================
setup.template.settings:
index.number_of_shards: 1
index.codec: best_compression
# =================================== Kibana ===================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
# Kibana Host
# Scheme and port can be left out and will be set to the default (http and 5601)
# In case you specify and additional path, the scheme is required: http://localhost:5601/path
# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
host: "https://<IP@>:5601"
username: "metricbeat_internal"
password: <pwd>
#ssl.enabled: true
# ------------------------------ Logstash Output -------------------------------
output.logstash:
# The Logstash hosts
hosts: ["<IP@>:5045"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
ssl.certificate_authorities: ["certs/ca.crt"]
# Certificate for SSL client authentication
#ssl.certificate:
# Client Certificate Key
#ssl.key:
username: "metricbeat_internal"
password: <pwd>
# # ================================= Processors =================================
# Configure processors to enhance or manipulate events generated by the beat.
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
# ================================== Logging ===================================
# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
logging.level: debug
logging.to_files: true
# ================================= ILM ======================================
setup.ilm.check_exists: false
- winlogbeat.yml :
# ======================== Winlogbeat specific options =========================
# event_logs specifies a list of event logs to monitor as well as any
# accompanying options. The YAML data type of event_logs is a list of
# dictionaries.
#
# The supported keys are name (required), tags, fields, fields_under_root,
# forwarded, ignore_older, level, event_id, provider, and include_xml. Please
# visit the documentation for the complete details of each option.
# https://go.es.io/WinlogbeatConfig
winlogbeat.event_logs:
- name: Application
ignore_older: 72h
- name: System
- name: Security
processors:
- script:
lang: javascript
id: security
file: ${path.home}/module/security/config/winlogbeat-security.js
- name: Microsoft-Windows-Sysmon/Operational
processors:
- script:
lang: javascript
id: sysmon
file: ${path.home}/module/sysmon/config/winlogbeat-sysmon.js
- name: Windows PowerShell
event_id: 400, 403, 600, 800
processors:
- script:
lang: javascript
id: powershell
file: ${path.home}/module/powershell/config/winlogbeat-powershell.js
- name: Microsoft-Windows-PowerShell/Operational
event_id: 4103, 4104, 4105, 4106
processors:
- script:
lang: javascript
id: powershell
file: ${path.home}/module/powershell/config/winlogbeat-powershell.js
- name: ForwardedEvents
tags: [forwarded]
processors:
- script:
when.equals.winlog.channel: Security
lang: javascript
id: security
file: ${path.home}/module/security/config/winlogbeat-security.js
- script:
when.equals.winlog.channel: Microsoft-Windows-Sysmon/Operational
lang: javascript
id: sysmon
file: ${path.home}/module/sysmon/config/winlogbeat-sysmon.js
- script:
when.equals.winlog.channel: Windows PowerShell
lang: javascript
id: powershell
file: ${path.home}/module/powershell/config/winlogbeat-powershell.js
- script:
when.equals.winlog.channel: Microsoft-Windows-PowerShell/Operational
lang: javascript
id: powershell
file: ${path.home}/module/powershell/config/winlogbeat-powershell.js
- name: HardwareEvents
- name: HP Sure Start
# ====================== Elasticsearch template settings =======================
setup.template.settings:
index.number_of_shards: 1
# =================================== Kibana ===================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
# Kibana Host
# Scheme and port can be left out and will be set to the default (http and 5601)
# In case you specify and additional path, the scheme is required: http://localhost:5601/path
# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
host: "https://IP@:5601"
username: "winlogbeat_internal"
password: <pwd>
# ------------------------------ Logstash Output -------------------------------
output.logstash:
# The Logstash hosts
hosts: ["IP@:5044"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
ssl.certificate_authorities: ["certs/ca.crt"]
# Certificate for SSL client authentication
username: "winlogbeat_internal"
password: <pwd>
# ================================= Processors =================================
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
# ================================= ILM ======================================
setup.ilm.enabled: auto
setup.ilm.rollover_alias: "winlogbeat"
setup.ilm.check_exists: false
setup.ilm.overwrite: true
- metricbeat/modules.d/system.yml :
# Module: system
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/7.10/metricbeat-module-system.html
- module: system
period: 10s
metricsets:
- cpu
#- load
- memory
#- network
- process
- process_summary
- socket_summary
#- entropy
#- core
- diskio
#- socket
#- service
#- users
process.include_top_n:
by_cpu: 5 # include top 5 processes by CPU
by_memory: 5 # include top 5 processes by memory
- module: system
period: 1m
metricsets:
- filesystem
- fsstat
processors:
- drop_event.when.regexp:
system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)'
- module: system
period: 15m
metricsets:
- uptime
- metricbeat/modules.d/windows.yml
# Module: windows
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/7.10/metricbeat-module-windows.html
- module: windows
metricsets:
- service
enabled: true
period: 1m
- module: windows
metricsets:
- perfmon
enabled: false
period: 10s
perfmon.ignore_non_existent_counters: false
perfmon.group_measurements_by_instance: false
perfmon.queries:
- object: "Process"
instance: ["svchost*", "conhost*"]
counters:
- name: "% Processor Time"
field: time.processor.pct
format: "float"
- name: "Thread Count"
field: thread_count
- name: "IO Read Operations/sec"
- object: "PhysicalDisk"
field : "disk"
instance: "*"
counters:
- name: "Disk Writes/sec"
- name: "% Disk Write Time"
field: "write_time"
format: "float"
- The metricbeat pipeline :
input {
beats {
port => 5045
ssl => true
ssl_key => "certs/logstash.pkcs8.key"
ssl_certificate => "certs/logstash.crt"
}
}
output {
elasticsearch {
hosts => ["https://IP@:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}"
cacert => "certs/ca.crt"
user => "logstash_internal"
password => "<pwd>"
}
}
- The winlogbeat pipeline :
input {
beats {
port => 5044
ssl => true
ssl_key => "certs/logstash.pkcs8.key"
ssl_certificate => "certs/logstash.crt"
}
}
output {
elasticsearch {
hosts => ["https://IP@:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}"
cacert => "certs/ca.crt"
user => "logstash_internal"
password => "<pwd>"
}
}