Filebeat to Logstash --> DBG [transport] handle error: EOF

Elastic Stack Beats
,
1

We have the following Filebeat 6.1.3 conf:

filebeat.prospectors:
  - type: log
    paths:
      - /usr/share/filebeat/taifunlogs/server.log

output.logstash:
  hosts: ["xxxxxx.com:443"]
  bulk_max_size: 999999999

logging.level: debug
logging.to_files: false
logging.to_syslog: false
loggins.metrice.enabled: false
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644
ssl.verification_mode: none

and Logstash 6.1.3 pipeline:

input {
http {
port => "5044"
}
}

Add your filters / logstash plugins configuration here

output {
elasticsearch {
hosts => ["https://376d1d6ba0b54ecf9ace06b3b8ddd4db.elasticsearch.xxxx.com"]
user => "es3iv3ARd9U6egMrg5"
password => "GeaUxFT5zmMJqUZT"
ssl => true
ssl_certificate_verification => true
codec => "plain"
workers => 1
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
manage_template => true
template_name => "logstash"
template_overwrite => true
}
}

We also configured a TCP routing from 443 to 5044, in fact we can run a test wget from the filebeat machine to the Logstash url (443) and the message is received correctly.
Unfortunately with this configuration this is what we get from filbeat (trying to inject a 173M message):

filebeat_1 | "message": "[#|2019-08-25T18:04:08.335+0200|FINE|Payara 4.1|ch.ergon.taifun.sql.DistributedWorksetSql|_ThreadID=263;_ThreadName=allegro/incrementalUpdateConsumer-managedThreadFactory-Thread-5;_TimeMillis=1566749048335;_LevelValue=500;ClassName=ch.ergon.taifun.base.jooq.impl.LoggingExecuteListener;MethodName=logFine;|Query with params : select CE_LOCK.LOCK_NAME, CE_LOCK.SUB_LOCK_NAME, CE_LOCK.LOCKING_FUNCTION, CE_LOCK.HOSTNAME, CE_LOCK.USERW, CE_LOCK.DATEW, CE_LOCK.USERI, CE_LOCK.DATEI from CE_LOCK where CE_LOCK.LOCK_NAME = 'REPORTING_SYNC_LOCK' for update|#]",
filebeat_1 | "prospector": {
filebeat_1 | "type": "log"
filebeat_1 | },
filebeat_1 | "beat": {
filebeat_1 | "name": "filebeat",
filebeat_1 | "hostname": "filebeat",
filebeat_1 | "version": "6.1.3"
filebeat_1 | }
filebeat_1 | }
filebeat_1 | 2019/08/26 09:11:33.420534 processor.go:275: DBG [publish] Publish event: {
filebeat_1 | "@timestamp": "2019-08-26T09:11:33.420Z",
filebeat_1 | "@metadata": {
filebeat_1 | "beat": "filebeat",
filebeat_1 | "type": "doc",
filebeat_1 | "version": "6.1.3"
filebeat_1 | },
filebeat_1 | "source": "/usr/share/filebeat/taifunlogs/server.log",
filebeat_1 | "offset": 2375631,
filebeat_1 | "message": "[#|2019-08-25T18:04:08.356+0200|INFO|Payara 4.1|ch.ergon.taifun.commonentities.lock.LockService|_ThreadID=263;_ThreadName=allegro/incrementalUpdateConsumer-managedThreadFactory-Thread-5;_TimeMillis=1566749048356;_LevelValue=800;|Acquired partial locks unsafely: REPORTING_INC_RUNNING(d-alg-be-11)|#]",
filebeat_1 | "prospector": {
filebeat_1 | "type": "log"
filebeat_1 | },
filebeat_1 | "beat": {
filebeat_1 | "version": "6.1.3",
filebeat_1 | "name": "filebeat",
filebeat_1 | "hostname": "filebeat"
filebeat_1 | }
filebeat_1 | }
filebeat_1 | 2019/08/26 09:11:33.420615 processor.go:275: DBG [publish] Publish event: {
filebeat_1 | "@timestamp": "2019-08-26T09:11:33.420Z",
filebeat_1 | "@metadata": {
filebeat_1 | "beat": "filebeat",
filebeat_1 | "type": "doc",
filebeat_1 | "version": "6.1.3"
filebeat_1 | },
filebeat_1 | 2019/08/26 09:12:05.648135 output.go:92: ERR Failed to publish events: client is not connected
filebeat_1 | 2019/08/26 09:12:05.648197 async.go:94: DBG [logstash] connect
filebeat_1 | 2019/08/26 09:12:05.648619 logger.go:22: INFO retryer: send unwait-signal to consumer
filebeat_1 | 2019/08/26 09:12:05.648644 logger.go:22: INFO done
filebeat_1 | 2019/08/26 09:12:05.648655 logger.go:22: INFO retryer: send wait signal to consumer
filebeat_1 | 2019/08/26 09:12:05.648665 logger.go:22: INFO done
filebeat_1 | 2019/08/26 09:12:05.654675 logger.go:22: INFO retryer: send unwait-signal to consumer
filebeat_1 | 2019/08/26 09:12:05.654698 logger.go:22: INFO done
filebeat_1 | 2019/08/26 09:12:05.658828 async.go:142: DBG [logstash] 295 events out of 295 events sent to logstash host xxxxxxxxx.com:443. Continue sending
filebeat_1 | 2019/08/26 09:12:05.660761 client.go:201: DBG [transport] handle error: EOF
filebeat_1 | 2019/08/26 09:12:05.660799 client.go:114: DBG [transport] closing
filebeat_1 | 2019/08/26 09:12:05.660869 async.go:235: ERR Failed to publish events caused by: EOF
filebeat_1 | 2019/08/26 09:12:05.689668 async.go:142: DBG [logstash] 1973 events out of 1973 events sent to logstash host xxxxxxxx.com:443. Continue sending
filebeat_1 | 2019/08/26 09:12:05.689696 async.go:99: DBG [logstash] close connection
filebeat_1 | 2019/08/26 09:12:05.689705 async.go:99: DBG [logstash] close connection
filebeat_1 | 2019/08/26 09:12:05.689724 async.go:235: ERR Failed to publish events caused by: client is not connected
filebeat_1 | 2019/08/26 09:12:05.689739 logger.go:22: INFO retryer: send wait signal to consumer

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.