Hey All,
I recently started using metricbeat on some remote servers and collecting the data on my central ES cluster. I'm sending the metricbeat data on each server using the logstash output to the remote ES server over port 5044.
server1 running metricbeat -> server2:5044 -> server2 running logstash -> localhost:9200
the problem I'm having is the ES index created looks like this:
%{[@metadata][beat]}-2017.09.29
Is this because the metadata on the central logstash server isn't recognized from the remote server sending the beat data?
Metricbeat.yml file on server1:
########################## Metricbeat Configuration ###########################
#============================ Config Reloading ===============================
# Config reloading allows to dynamically load modules. Each file which is
# monitored must contain one or multiple modules as a list.
metricbeat.config.modules:
# Glob pattern for configuration reloading
path: ${path.config}/conf.d/*.yml
# Period on which files under path should be checked for chagnes
reload.period: 10s
# Set to true to enable config reloading
reload.enabled: false
#========================== Modules configuration ============================
metricbeat.modules:
#------------------------------- System Module -------------------------------
- module: system
metricsets:
# CPU stats
- cpu
# System Load stats
- load
# Per CPU core stats
- core
# IO stats
- diskio
# Per filesystem stats
- filesystem
# File system summary stats
- fsstat
# Memory stats
- memory
# Network stats
- network
# Per process stats
- process
# Sockets and connection info (linux only)
- socket
enabled: true
period: 10s
processes: ['.*']
# if true, exports the CPU usage in ticks, together with the percentage values
#cpu_ticks: false
# A list of filesystem types to ignore. The filesystem metricset will not
# collect data from filesystems matching any of the specified types, and
# fsstats will not include data from these filesystems in its summary stats.
#filesystem.ignore_types: []
# Enable collection of cgroup metrics from processes on Linux.
#process.cgroups.enabled: true
# Configure reverse DNS lookup on remote IP addresses in the socket metricset.
#socket.reverse_lookup.enabled: false
#socket.reverse_lookup.success_ttl: 60s
#socket.reverse_lookup.failure_ttl: 60s
#------------------------------- Apache Module -------------------------------
- module: apache
metricsets: ["status"]
enabled: true
period: 10s
# Apache hosts
hosts: ["http://127.0.0.1"]
# Path to server status. Default server-status
#server_status_path: "server-status"
# Username of hosts. Empty by default
#username: test
# Password of hosts. Empty by default
#password: test123
#-------------------------------- MySQL Module -------------------------------
- module: mysql
metricsets: ["status"]
enabled: true
period: 10s
# Host DSN should be defined as "user:pass@tcp(127.0.0.1:3306)/"
# The username and password can either be set in the DSN or using the username
# and password config options. Those specified in the DSN take precedence.
hosts: ["metricbeat:hyTY@45gbcvxde!2@tcp(127.0.0.1:3306)/"]
# Username of hosts. Empty by default.
#username: root
# Password of hosts. Empty by default.
#password: secret
# By setting raw to true, all raw fields from the status metricset will be added to the event.
#raw: false
#================================ Outputs ======================================
# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.
#----------------------------- Logstash output ---------------------------------
output.logstash:
# Boolean flag to enable or disable the output module.
#enabled: true
# The Logstash hosts
hosts: ["xx.xx.xx.xx:5044"]
# Number of workers per Logstash host.
#worker: 1
# Set gzip compression level.
#compression_level: 3
# Optional load balance the events between the Logstash hosts
#loadbalance: true
# Number of batches to be send asynchronously to logstash while processing
# new batches.
#pipelining: 0
# Optional index name. The default index name is set to name of the beat
# in all lowercase.
#index: 'metricbeat'
# SOCKS5 proxy server URL
#proxy_url: socks5://user:password@socks5-server:2233
# Resolve names locally when using a proxy server. Defaults to false.
#proxy_use_local_resolver: false
# Enable SSL support. SSL is automatically enabled, if any SSL setting is set.
#ssl.enabled: true
# Configure SSL verification mode. If `none` is configured, all server hosts
# and certificates will be accepted. In this mode, SSL based connections are
# susceptible to man-in-the-middle attacks. Use only for testing. Default is
# `full`.
#ssl.verification_mode: full
# List of supported/valid TLS versions. By default all TLS versions 1.0 up to
# 1.2 are enabled.
#ssl.supported_protocols: [TLSv1.0, TLSv1.1, TLSv1.2]
# Optional SSL configuration options. SSL is off by default.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"
# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"
# Optional passphrase for decrypting the Certificate Key.
#ssl.key_passphrase: ''
# Configure cipher suites to be used for SSL connections
#ssl.cipher_suites: []
# Configure curve types for ECDHE based cipher suites
#ssl.curve_types: []
Here is the logstash conf file for server2:
input {
beats {
port => 5044
}
}
# The filter part of this file is commented out to indicate that it is
# optional.
# filter {
#
# }
output {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
Any suggestions on how to set this up in sich a way that
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
is converted as expected?
Thank you!