Hello,
I try to push some log files from filebeat to logstash.
Everything is working from the server A (filebeat) to Logstash
Then I tried to push same logs from server B (filebeat) to Logstash and I have these errors :
2019-10-01T11:04:26.802+0200 INFO pipeline/output.go:95 Connecting to backoff(async(tcp://10.118.120.65:1762))
2019-10-01T11:04:26.806+0200 INFO pipeline/output.go:105 Connection to backoff(async(tcp://10.118.120.65:1762)) established
2019-10-01T11:04:53.764+0200 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":10,"time":{"ms":13}},"total":{"ticks":180,"time":{"ms":184},"value":0},"user":{"ticks":170,"time":{"ms":171}}},"handles":{"limit":{"hard":65535,"soft":65535},"open":8},"info":{"ephemeral_id":"26170285-e93b-4c2b-a210-5ba6fc1f7576","uptime":{"ms":30018}},"memstats":{"gc_next":33498480,"memory_alloc":16922056,"memory_total":40137952,"rss":44019712},"runtime":{"goroutines":25}},"filebeat":{"events":{"active":4117,"added":4119,"done":2},"harvester":{"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"events":{"active":2048,"batches":1,"total":2048},"type":"logstash"},"pipeline":{"clients":1,"events":{"active":4117,"filtered":2,"published":4116,"retry":2048,"total":4119}}},"registrar":{"states":{"current":1,"update":2},"writes":{"success":2,"total":2}},"system":{"cpu":{"cores":2},"load":{"1":0,"15":0.05,"5":0,"norm":{"1":0,"15":0.025,"5":0}}}}}}
2019-10-01T11:04:56.846+0200 ERROR logstash/async.go:256 Failed to publish events caused by: write tcp 10.117.56.140:26856->10.118.120.65:1762: i/o timeout
2019-10-01T11:04:58.341+0200 ERROR pipeline/output.go:121 Failed to publish events: write tcp 10.117.56.140:26856->10.118.120.65:1762: i/o timeout
2019-10-01T11:04:58.341+0200 INFO pipeline/output.go:95 Connecting to backoff(async(tcp://10.118.120.65:1762))
2019-10-01T11:04:58.345+0200 INFO pipeline/output.go:105 Connection to backoff(async(tcp://10.118.120.65:1762)) established
A telnet from the server B to Logstash is OK and if I try to push some random data to logstash in the telnet session, I obtain some logs to the logstash side :
[2019-10-01T11:19:56,575][INFO ][org.logstash.beats.BeatsHandler] [local: 10.118.120.65:1762, remote: 10.117.56.140:27276] Handling exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 117
[2019-10-01T11:19:56,576][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
Logstash and Filebeat are the same version (7.2.0)
The filebeat configuration between Server A and Server B is exactly the same (copy paste from A to B)
Here is the filebeat conf :
# =========================== Filebeat inputs =============================
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/opt/data/flat/squid/log/daily-access.log
fields:
logtype: squid_access_log_dev
fields_under_root: true
#============================= Filebeat modules ===============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# Period on which files under path should be checked for changes
#reload.period: 10s
#==================== Elasticsearch template setting ==========================
setup.template.settings:
index.number_of_shards: 1
#index.codec: best_compression
#_source.enabled: false
#================================ General =====================================
# ...
#================================ Outputs =====================================
# Configure what output to use when sending the data collected by the beat.
#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
# Array of hosts to connect to.
# hosts: ["localhost:9200"]
# Optional protocol and basic auth credentials.
#protocol: "https"
#username: "elastic"
#password: "changeme"
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["10.118.120.65:1762"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"
# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"
#================================ Processors =====================================
# Configure processors to enhance or manipulate events generated by the beat.
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
#================================ Logging =====================================
#...
#================================= Migration ==================================
#...
Logstash conf is pretty simple too :
input {
beats {
port => "1762"
}
}
filter {
dissect {
mapping => {
"message" => "%{month->} %{} %{} %{host} %{} %{timestamp->} %{} %{user_id} %{req_status} %{status_code} %{user_ip} %{user_req_size} %{method} %{url} %{mime_type} %{reply_size} %{hierarchy} %{fwd_ip} %{srv_ip} %{srv_port} %{}"
}
convert_datatype => {
"status_code" => "int"
"user_req_size" => "int"
"reply_size" => "int"
"srv_port" => "int"
}
}
grok {
match => {
"message" => "%{SYSLOGTIMESTAMP:syslog_timestamp}.*%{NUMBER:duration:int}"
}
#remove_field => ["message"]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss" ]
timezone => "Europe/Paris"
}
}
output {
elasticsearch {
hosts => ["10.118.123.226:1761", "10.118.123.227:1761", "10.118.123.229:1761"]
index => "squid-%{+YYYY.MM.dd}"
manage_template => true
template => "/etc/logstash/conf.d/squid_access_log_mapping.json"
template_name => "squid_template"
}
}
Any idea ?
Thank you
