When connecting metricbeats to logstash, I'm metricbeats is unable to connect:
"Caused by: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received"
We have EL5.6+Xpack , Kib5.6 +Xpack (NonSSL passwords for EL5.6)
-
I installed Metricbeat 6.1.1 (with a password and hooked it into ES with no problems.
-
I installed Logstash 5.x and upgraded it to EL 6.x (with a username + password), and it connected to EL fine. (using http inputs). I updated the logstash-input-beats.
-
When I change Logstash to "input { beats { port => "5044" } }". And repointed MetricBeat (from EL to Logstash (with and without the password in the now-defunct EL stanza), I get:
"Caused by: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received"
From MetricBeat I get a connection reset by peer.
===
Metric Beat config
egrep -v "^[ ]#|^$" /etc/metricbeat/metricbeat.yml
metricbeat.config.modules:
path: ${path.config}/modules.d/.yml
reload.enabled: true
reload.period: 10s
setup.template.settings:
index.number_of_shards: 1
index.codec: best_compression
setup.kibana:
host: "localhost:5601"
output.elasticsearch:
hosts: ["localhost:5044", "10.x.y.z:5044"]
logging.level: debug
========
input {
beats {
port => "5044"
}
}
filter {
grok {
match => {"message" => [ "hdfs%{GREEDYDATA:datanode_process}org.apache.hadoop.hdfs.server.datanode.DataNode",
"##REAL##%{GREEDYDATA:real_message}",
"%{GREEDYDATA:basic_message}"]}
}
if [datanode_process] {
mutate {
replace => { "message" => "%{datanode_process}" }
add_tag => ["DataNode"]
}
} else if [real_message] {
mutate {
replace => { "message" => "%{real_message}" }
add_tag => ["Real"]
}
} else {
mutate {
add_tag => ["Basic"]
}
}
}
output {
elasticsearch {
hosts => ["xxx:9200", "yyy:9200", "zzz:9200"]
user => username
password => pass
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}