kvaga
January 18, 2019, 11:53am
1
client - filebeat-6.5.4-1.x86_64
logs from client
2019-01-18T13:56:20.033+0300 ERROR logstash/async.go:256 Failed to publish events caused by: write tcp 10.129.10.8:34714->10.129.10.7:5044: write: connection reset by peer
2019-01-18T13:56:20.033+0300 DEBUG [logstash] logstash/async.go:116 close connection
2019-01-18T13:56:21.033+0300 ERROR pipeline/output.go:121 Failed to publish events: write tcp 10.129.10.8:34714->10.129.10.7:5044: write: connection reset by peer
2019-01-18T13:56:21.033+0300 INFO pipeline/output.go:95 Connecting to backoff(async(tcp://10.129.10.7:5044))
2019-01-18T13:56:21.033+0300 DEBUG [logstash] logstash/async.go:111 connect
2019-01-18T13:56:21.035+0300 INFO pipeline/output.go:105 Connection to backoff(async(tcp://10.129.10.7:5044)) established
2019-01-18T13:56:21.035+0300 DEBUG [logstash] logstash/async.go:159 3 events out of 3 events sent to logstash host 10.129.10.7:5044. Continue sending
logs from LS server
[2019-01-18T13:56:21,035][DEBUG][org.logstash.beats.ConnectionHandler] 1c400199: batches pending: true@id = "plain_f6bf0dc8-6c79-448d-9431-02e244c9471a"@enable_metric = true@charset = "UTF-8"
no errors when logstash starting
Tek_Chand
January 18, 2019, 12:18pm
2
@kvaga ,filebeat.yml file configuration.
Thanks.
kvaga
January 18, 2019, 12:23pm
3
filebeat.yml
#=========================== Filebeat inputs =============================
filebeat.inputs:
- type: log
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /opt/app/log/tomcat/access*.log
#============================= Filebeat modules ===============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
#==================== Elasticsearch template setting ==========================
setup.template.settings:
index.number_of_shards: 3
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["10.129.10.7:5044"]
#================================ Procesors =====================================
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
Tek_Chand
January 18, 2019, 12:29pm
4
@kvaga ,
Your configuration seems fine. Please check the connectivity from your filebeat system to logstash server using below command:
telnet your_logstash_server_ip 5044
Also try to increase the client_inactivity_timeout in your logstash.
Thanks.
kvaga
January 18, 2019, 12:34pm
5
#telnet kibana 5044
logstash.conf
input {
beats {
client_inactivity_timeout => 3000
port => 5044
ssl => false
}
}
watching...
kvaga
January 18, 2019, 12:54pm
6
now error is gone (i hope)
how can i copy config filebeat 5.5 to 6.5 version ?
prospectors:
-
paths:
- /opt/app/log/tomcat/access*.log
input_type: log
document_type: access_log_stage
-
paths:
- /opt/app/log/tomcat/console*.log
input_type: log
document_type: console_log_stage
multiline:
pattern: '^*\|[[:space:]]*at|^*\|[[:space:]]Caused by:|^*\|[[:space:]]*\.\.\.[[:space:]]'
negate: false
match: after
max_lines: 500
timeout: 5s
input_type log -> - type: log
document_type:
Tek_Chand
January 21, 2019, 4:55am
8
@kvaga ,
You can uninstall filebeat5.5 and install filebeat6.5 again. Its better way may be some changes in configuration of 5.5 and 6.5.
Multiline are used when you logs are in multiline and want to manage them (treat as single line log).
I think document_type is deprecated in Filebeat6.5. Please refer the below link with example.
elasticsearch, filebeat
Thanks.
kvaga
January 21, 2019, 10:54am
9
now my filebeat.yml inputs
#=========================== Filebeat inputs =============================
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
- type: log
# Change to true to enable this input configuration.
enabled: false
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /opt/app/log/tomcat/access*.log
#- c:\programdata\elasticsearch\logs\*
fields:
service: access_log_prestage
and now when I`m tryng to start filebeats got error
019-01-21T13:53:37.991+0300 ERROR instance/beat.go:800 Exiting: error initializing publisher: missing required field accessing 'output.logstash.hosts'
Exiting: error initializing publisher: missing required field accessing 'output.logstash.hosts'
but in filebeat.yml is
#----------------------------- Logstash output --------------------------------
The Logstash hosts
hosts: ["10.129.10.7:5044"]
Tek_Chand
January 21, 2019, 10:57am
10
@kvaga ,
kvaga:
enabled: false
You need to set it to true and its showing false in above configuration:
enable: true
Tek_Chand
January 21, 2019, 11:07am
12
@kvaga ,
This error is due to some indentation issue in your output.logstash part.
Please verify it again. Its should be look like below:
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["your_logstash_server_ip:5044"]
bulk_max_size: 1024
Thanks.
kvaga
January 21, 2019, 11:12am
13
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["10.129.10.7:5044"]
bulk_max_size: 1024
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"
# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"
#================================ Procesors =====================================
start log:
2019-01-21T14:11:22.006+0300 INFO instance/beat.go:278 Setup Beat: filebeat; Version: 6.5.4http://metadata.tencentyun.com/meta-data/instance-id: dial tcp: lookup metadata.tencentyun.com on 8.8.8.8:53: no such host, metadata={}]http://169.254.169.254/metadata/instance/compute?api-version=2017-04-02: dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
Tek_Chand
January 21, 2019, 11:22am
14
Your 2nd and 3rd line also are also starting from same position as first one.
But it should be look like below:
output.logstash:
hosts: ["elk.promobitech.com:5044"]
bulk_max_size: 1024
Give two spacebar space in 2nd and 3rd line. I have tested it at my end.
Its should fix your issue.
Thanks.
Tek_Chand
January 21, 2019, 11:26am
16
@kvaga ,
kvaga:
yes ! filebeat started !
Glad to hear that your problem is fixed. Please mark it as solution if it resolve your issue.
Thanks.
system
February 18, 2019, 11:26am
17
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
